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training  tasks. — 

^Computer  speech  recognition  and  generation  were  characterized  as  rapidly 
advancing  technologies  that  are  ready  now  for  application  in  automated  training 
systems  - 


A  human  factors  perspective  was  advanced  by  emphasizing  the  importance 
of  the  trainee  in  the  design  of  complex  automated  training  systems.  Training 
effectiveness  and  user  acceptance  are  objectives  to  be  sought  through  careful 
design  of  the  Voice  Subsystem  and  supporting  models  and  subsystems  described 
In  the  report. 

>rBoth  complex  and  simple  applications  of  AST  for  training  were  addressed. 
The  emphasis  was  on  complex  systems  designed  to  reduce  the  need  for  instructors 
and  other  training  support  personnel.  This  reduction  can  be  accomplished 
through  speech  interactive  simulation  and  the  automation  of  performance 
measurement,  curriculum  control,  record  keeping,  and  instructional  presentation 


•ICUMtTV  CkAUmCATlON  or  this  PA«C(«to«  Om 


NAVTRAEQUIPCEN  80-C-0057-1 


FOREWORD 


This  report  is  part  of  work  done  under  Project  3753,  Application  of  Voice 
Technology  in  Automated  Systems.  The  objective  was  to  develop  techniques  to 
augment  training  of  personnel  for  jobs  such  as  Air  Intercept  Controller. 


Surface-based  controller  tasks  require,  in  addition  to  an  instructor, 
a  keyset  operator  who  functions  as  a  "pseudo  pilot”  to  maneuver  the  simulated 
aircraft  in  response  to  control  commands  from  the  student.  The  job  is  very 
boring  and  performance  soon  degrades.  Further,  training  cannot  be  conducted 
during  extended  or  after  normal  working  hours  due  to  nonavailability  of  keyset 
operators.  Variance  introduced  by  the  use  of  human  "simulator  pilots"  is  so 
contaminating  that  proper  training  control  and  evaluation  is  extremely  diffi¬ 
cult.  Integration  of  speech  technology  with  automated  adaptive  training 
systems  offers  a  potential  solution  to  this  problem  by  providing  automation 
of  "simulator  pilots"  capable  of  consistant  performance  with  full  time  avail¬ 
ability. 

The  application  of  computer  speech  recognition  and  generation  is  a  rapidly- 
developing  technology  which  is  constantly  being  improved.  This  project  has 
spawned  the  integration  of  this  technology  into  three  Navy  prototype  training 
systems:  (1)  an  experimental  Ground  Controlled  Approach  (GCA)  controller 

training  system,  (2)  an  Air  Intercept  Controller  (AIC)  controller  training 
system,  and  (3)  a  Landing  Signal  Officer  (LSO)  training  system  (Project  7754) . 


Automated  speech  technology  should  also  be  readily  transferable  to  other 
military  and  civilian  applications.  To  facilitate  the  design  of  such  systems. 


this  phase  of  Project  3753  was  aimed  at  aenerating  miidelines  for  applications 
of  speech  technology  in  automated  training  systems.  The  design  guides  are 
primarily  intended  for  instructional  developers  and  training  device  design 
engineers  who  may  be  interested  in  the  use  of  computer  speech  technology  in 

their  training  systems.  These  guides  should  be  considered  tentative,  howevei 
until  they  are  validated. 
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PREFACE 


This  report  describes  the  work  performed  under  NAVTRAEQUIPCEN  Contract 
N61339-80-C-0057.  Dr.  Robert  Breaux  served  as  the  NAVTRAEQUIPCEN  Project 
Engineer. 

The  authors  would  like  to  thank  the  key  scientists  who  consented  to  be 
interviewed  In  the  course  of  the  project.  In  particular,  Logicon,  Inc. 
generously  authorized  interviews  with  personnel  involved  in  the  development 
of  prototype  training  systems  that  included  automated  speech  technology. 
Dr.  Douglas  C.  Chatfield,  President  of  Behavioral  Evaluation  and  Training 
Systems  (BETS),  was  invaluable  as  a  consultant  on  the  project.  These 
individuals,  however,  should  not  be  held  responsible  for  the  content  of  the 
report,  which  reflects  solely  the  interpretations  and  opinions  of  the 
authors. 

We  appreciate  the  tireless  work  of  Rosemary  Wescott  in  the  preparation 
and  production  of  the  report. 

The  design  guides,  found  in  Appendices  A  -  D,  were  written  by  Mr.  John 
Cotton.  The  body  of  the  report  was  written  by  Dr.  Michael  E.  McCauley. 
Any  redundancy  between  these  presentations  was  considered  beneficial. 
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SECTION  I 
INTRODUCTION 


Speech  interactive  systems  provide  some  direct  benefits  over  more 
traditional  man-computer  interfaces  such  as  the  keyboard.  Speech  is  a 
natural  communication  medium.  It  eliminates  the  requirement  for  naive  or 
temporary  users  to  be  trained  on  keyboard  functions.  It  can  be 
particularly  effective  in  a  task  situation  charactertized  as  "hands-busy, 
eyes-busy."  Speech  does  not  compete  directly  with  a  visual -manual  task;  a 
keyboard  does.  A  speech  interactive  system  also  can  provide  some  mobility 
to  the  user,  while  keyboard/CRT  interface  fixes  the  user  in  one  location. 

In  training  applications,  automated  speech  technology  (AST)  promises 
to  reduce  or  eliminate  instructor  workload  for  verbal  tasks,  such  as  air 
traffic  control  (ATC).  It  promises  to  eliminate  the  need  for  instructional 
support  personnel,  such  as  pseudo-pilots,  who  listen  to  an  ATC  trainee's 
verbal  transmissions  and  simulate  the  pilot  response.  These  capabilities 
enable  training  to  be  automated,  producing  significant  savings  in  personnel 
costs. 

BACKGROUND 

A  series  of  research  and  development  (RAD)  programs  within  the  Navy 
has  been  directed  toward  the  development  of  experimental  prototype  training 
systems  to  investigate  the  feasibility  and  effectiveness  of  applying 
automated  speech  technology  to  training  systems.  Specifically,  three  R&D 
efforts  are  seeking  to  provide  automated  training  for  Precision  Approach 
Radar  Controllers  (PAR),  Air  Intercept  Controllers  (AIC),  and  Landing 
Signal  Officers  (LSO).  These  systems  are  characterized  by  an  automated 
adaptive  syllabus,  simulated  instructor,  simulated  environmental  events, 
automated  performance  measurement,  and  a  strong  voice  interaction  between 
the  trainee  and  the  system. 

Both  successes  and  difficulties  have  been  encountered  during  these 
prototype  and  development  programs.  A  need  existed  to  review  these  prior 
developments  and  to  produce  guidelines  for  the  future  design  of  AST  in  Navy 
training  systems.  The  purpose  of  the  present  study  was  to  integrate 
current  concepts  in  training  technology  with  the  lessons  learned  from  these 
RAD  programs  in  order  to  produce  system  design  guides  which  will  provide 
direction  for  the  functional  design  of  future  AST  training  systems. 

The  design  guides  produced  by  this  study  will  be  directed  toward 
training  analysts,  subject-matter-experts,  and  systems  analysts  to  channel 
and  facilitate  their  joint  efforts  in  developing  detailed  functional 
specifications  for  their  particular  training  application.  Additionally,  an 
evaluation  plan  will  be  generated  to  provide  a  mechanism  for  testing, 
evaluating  and  validating  the  experimental  design  guides. 
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SPEECH  TECHNOLOGY  EMPHASIS 

The  emphasis  in  this  work  has  been  on  AST.  This  term  is  used  to  refer 
to  both  computer  speech  recognition  and  generation.  The  human-computer 
dialogue  enabled  by  AST  has  the  potential  for  revolutionizing  the  use  of 
computers  in  society  (see  Evans,  1979;  and  Toffler,  1980).  NAVTRAEQUIPCEN 
has  been  actively  pursuing  the  practical  applications  of  this  technology 
through  the  development  of  prototype  speech- interactive  training  systems. 
Like  any  new  technology  in  its  first  applications,  some  shortcomings  have 
been  encountered  in  the  prototype  systems.  One  purpose  of  this  report  is 
to  serve  as  the  institutional  memory  of  these  shortcomings  but,  more 
importantly,  to  suggest  ways  to  minimize  the  difficulties  in  future 
appl ications. 

Although  the  emphasis  of  this  study  is  AST,  related  topics  are 
included,  such  as  automated  instructor  models,  simulation  control, 
instructional  systems  development  (ISO),  and  automated  training  system 
integration.  A  schematic  diagram  of  the  contents  of  this  report  is  given 
in  Figure  1. 

The  present  study  was  directed  specifically  toward  Navy  training 
systems,  but  much  of  the  information  is  applicable  to  other  types  of 
systems  that  include  human-computer  interaction  via  voice. 

SCOPE  OF  THE  DESIGN  GUIDES 

The  design  guides  are  intended  as  a  primer  on  AST  applications  to 
training  system  design.  They  are  aimed  at  experienced  training  system 
designers  who  are  unfamiliar  with  the  capabilities  and  limitations  of  this 
new  technology. 

The  design  guides  are  necessarily  general,  in  order  to  be  useful  in  a 
variety  of  training  system  applications.  Specific  examples  of  training 
systems  are  given  to  assist  in  conceptualizing  applications  issues. 

The  design  guides  are  directed  toward  the  FUNCTIONAL  requirements  of 
AST  training  systems.  To  go  beyond  functional  requirements  into 
engineering  design  specifications  would  be  impossible  within  the  framework 
of  generic  design  guidelines.  For  this  reason,  and  because  the  technology 
of  computer  speech  recognition  is  evolving  rapidly,  manufacturer's  names 
and  product  specifications  are  not  included  in  the  design  guides. 

Suggestions  are  given  for  how  to  use  a  speech  interactive  capability 
in  complex,  intermediate,  and  simple  training  systems.  An  example  of  a 
complex  system  is  one  that  includes  the  following:  real-time  speech 
control  in  simulation  (as  in  a  ground-controlled  approach);  speech 
generation;  automated  instruction;  automated  performance  measurement;  and 
automated- adapt i ve  curriculum  control.  The  capabilities  and  limitations  of 
AST  must  be  considered  in  the  design  of  all  the  automated  subsystems  in 
such  a  complex  system.  An  example  of  an  intermediate  system  is  to  replace 
an  existing  pseudo  pilot  with  voice  interactions  between  the  trainee  and  a 
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Section  1 


Figure  1.  Voice  Technology  Final  Report  Layout 
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pilot/aircraft  model,  but  to  refrain  from  automating  instructor  functions. 
An  example  of  a  simple  system  is  to  replace  a  keyboard  with 
voice-interaction  in  a  computer-assisted  instruction  (CAI)  system. 

In  addition  to  discussing  various  levels  of  complexity  in  AST  training 
applications,  the  design  guides  suggest  how  to  match  speech  system 
capabilities  with  training  requirements,  and,  more  generally,  how  to 
incorporate  AST  into  the  system  development  cycle.  The  benefit  from  the 
design  guides  herein  will  continue  to  occur  up  to  the  point  when  the  system 
engineers  write  the  detailed  specification  for  a  training  system. 

A  brief  overview  is  given  of  the  current  and  projected  capabilities  of 
speech  technology,  but,  the  design  guides  focus  on  the  principles  of 
application  of  AST  to  training  systems,  rather  than  on  a  detailed 
technology  review. 
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SECTION  II 
METHODS 


SUMMARY 


The  methods  of  gathering  information  in  support  of  the  design  guide 
development  included:  1)  Navy  AST  training  system  document  review;  2) 
observation  of  Navy  AST  applications;  3)  literature  search  and  review;  and 
4)  interviews  with  key  scientists.  Using  these  methods,  information  was 
obtained  on: 


o  Computer  speech  recognition  technology 
o  Computer  speech  generation  technology 
o  Navy  AST  training  systems 

Precision  Approach  Radar  Training  System 
Air  Intercept  Controller  Training  System 
Landing  Signal  Officer  Training  System 
o  Automated  training  systems  design 
o  Instructor  modeling 
o  Automated  performance  measurement 
o  Artificial  intelligence  (AI)  and  AST 
o  Other  Navy,  Air  Force  and  FAA  applications  of  AST 


The  information  was  collated,  organized  and  analyzed  for  applicability 
to  AST  training  system  development.  The  project  Interim  Report  and  Interim 
Meeting  provided  a  forum  for  information  presentation  and  feedback  from 
NAVTRAEQUIPCEN.  In  response  to  the  feedback,  subsequent  work  broadened  to 
include  AST  applications  other  than  complex,  totally  automated  training 
systems.  Nevertheless,  the  focus  remained  on  the  integration  of  AST  with 
other  advanced  training  concepts,  such  as  automated  instructor  and  trainee 
models  in  a  real-time,  voice  interactive  simulation. 

NAVY  PROTOTYPE  AST  TRAINING  SYSTEM  DOCUMENT  REVIEW 

An  important  source  of  information  for  the  project  was  the  set  of 
technical  reports  which  document  the  development  and  use  of  AST  in  Navy 
prototype  training  systems,  specifically,  the  Precision  Approach  Radar 
Training  System  (PARTS;  also  known  as  the  Ground  Controlled  Approach 
Controller  Training  System,  GCA-CTS),  the  Prototype  Automated  Controller 
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Training  System  for  Air  Intercept  Controllers  (PACTS-AIC;  also  known  as  Air 
Controller  Exerciser,  ACE),  and  the  Landing  Signal  Officer  Training  System 
(LSOTS). 

The  documentation  of  these  systems  is  available  in  NAVTRAEQUIPCEN 
technical  reports-  A  listing  of  the  reports  obtained  and  reviewed  during 
this  project  is  given  in  the  Bibliography.  A  substantial  amount  of  the 
documentation  was  not  directly  applicable  to  the  present  study  because  it 
contained  extensive  description  of  system  software.  The  most  valuable 
information  for  the  present  project  was  obtained  from  the  reports  dealing 
with  behavioral  objectives,  functional  requirements,  training 
characteristics,  functional  design,  and  training  effectiveness. 

OBSERVATION  OF  NAVY  AST  TRAINING  SYSTEMS 

Both  PARTS  and  ACE  have  been  observed  directly  by  Canyon  personnel 
during  test  conditions  with  actual  Navy  trainees.  These  observations 
occurred  at  the  Navy  Air  Traffic  Control  School  in  Memphis,  TN,  and  at  the 
Fleet  Combat  Training  Center  Pacific,  in  San  Diego,  CA.  These  observations 
occurred  during  training  effectiveness  evaluations  conducted  by  Canyon 
Research  Group,  Inc.  for  NAVTRAEQUIPCEN.  (McCauley  and  Semple,  1980; 
McCauley,  Root,  and  Muckier,  in  press)  The  "hands  on"  experience  provided 
substantial  insight  into  the  strengths  and  weaknesses  of  these  early 
efforts  to  develop  a  fully  automated  training  system  around  speech 
technology. 

LITERATURE  SEARCH  AND  REVIEW 

A  literature  search  was  performed  to  identify  articles  relevant  to  the 
project.  The  Lockheed  Dialog  system  was  used  for  the  automated  literatuare 
search.  Six  data  bases  were  searched,  using  combinations  of  the  key  words 
"comou ter/ automated  ,  speech/voi ce/1 anguage,  recognition/understand¬ 
ing/  synthes i s/genera t ion/ technol ogy,  interactive  training/i nst ruction 
system."  A  total  of  224  citations  was  screened  by  titles  and  approximately 
30  were  obtained  and  reviewed.  Additionally,  secondary  references  and  a 
"manual"  literature  search  netted  approximately  30  more  pertinent  articles 
and  reports.  These  citations  were  listed  exhaustively  in  the  Interim 
Report  for  this  project. 

Many  of  the  references  obtained  and  reviewed  during  the  project  are 
cited  in  the  Bibliography.  It  represents  a  relatively  comprehensive 
compilation  of  the  scientific  and  technical  literature  pertaining  to  AST 
and  its  application  to  training  systems,  as  of  1981. 

INTERVIEWS  WITH  KEY  SCIENTISTS 

Interviews  were  conducted  with  key  scientists  involved  in  research  and 
development  of  AST  applications.  Most  of  the  interviews  followed  a 
structured  outline  and  were  done  in-person.  Some  less  extensive  interviews 
were  done  by  telephone. 
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The  interviews  supplemented  the  information  obtained  from  the  training 
system  documentation  and  literature  reviews.  A  list  of  the  people 
interviewed  follows: 


NAME 

ORGANIZATION 

M.  Grady 

Logicon,  Inc. 

M.  Hicklin 

Logicon,  Inc. 

E.  Butler 

Logicon,  Inc. 

R.  Halley 

Logicon,  Inc. 

G.  Slemon 

Logicon,  Inc. 

D.  Chatfield 

Behavioral  Evaluation 
Training  Systems,  Inc. 

and 

R.  Lynchard 

Eagle  Technology,  Inc. 

C.  Coler 

NASA 

D.  Connolly 

FAA 

D.  Lambert 

NOSC 

G.  Poock 

NPG  School 

M.  Strieb 

Analytics,  Inc. 

E.  Werkowitz 

USAF  FIGR 

Telephone  interviews  were 

obtained  with: 

A.  Craft 

U.S.  Postal  Service 

G.  White 

Threshold  Technology, 

Inc 

J.  Welch 

Threshold  Technology, 

Inc 

The  key  scientist  interviews  proved  to  be  a  rich  source  of  information 
on  AST  applications.  Each  application  has  a  unique  set  of  problems  to  be 
solved.  A  wide  range  of  performance  was  obtained  from  the  various  speech 
recognizer  applications.  The  information  obtained  through  the  interviews 
indicated  a  number  of  variables  that  influence  speech  recognition 
performance,  such  as  speaker  characteristics,  noise  environment,  speech 
sampling  procedures,  vocabulary  characteristics,  and  the  degree  of  stress 
on  the  user.  The  interviews  were  valuable  in  identifying  problem  areas 
rather  than  solving  the  problems. 
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SECTION  III 

SPEECH  TECHNOLOGY,  NOW  AND  IN  THE  FUTURE 


A  brief  overview  of  AST  will  be  given,  emphasizing  its  present 
capabilities  and  limitations.  The  overview  will  be  followed  by  discussions 
of  AST  applications  to  training  systems  in  Sections  IV  and  V. 

A  comprehensive  review  of  the  history  of  automated  speech  recognition 
is  beyond  the  scope  of  this  report.  Reviews  and  historical  perspectives 
are  available  in  Breaux,  Curran,  and  Huff,  (1978),  Dixon  and  Martin, 

(1979),  Hill,  (1971),  Lea,  (1980),  Lindgren,  (1965),  Martin  (1977),  and 

Reddy,  (1976). 

Likewise,  discussion  of  the  machine  processes,  algorithms,  and 

strategies  for  recognizing  speech  are  beyond  the  scope  of  the  present 

report.  Collections  of  articles  on  these  topics  can  be  found  in  Dixon  and 
Martin,  (1979)  and  Lea  (1980). 

SPEECH  RECOGNITION  AND  UNDERSTANDING 

Automated  speech  recognition,  according  to  Lea  (1980,  p.  39),  has  a 
thirty  year  history  "speckled  with  limited  successes  and  repetitive 
discoveries  of  old  ideas,  and  yet  with  a  growing  ability  to  successfully 
handle  small  vocabularies  of  words  spoken  in  isolation." 

In  the  1930s  and  1940s  the  evolving  technology  of  radio  gave  rise  to 
the  sound  spectrograph,  a  visual  depiction  of  the  acoustical  energy 
associated  with  speech.  The  parameters  associated  with  the  spectrogram, 
energy  by  frequency  by  time,  became  the  building  blocks  for  subsequent 
efforts  at  automated  speech  recognition.  In  the  1950s  researchers  at  the 
Bell  Telephone  Laboratories  developed  a  system  that  successfully 
"recognized"  a  limited  number  of  stored  word  patterns  spoken  by  a  single 
speaker  (Dudley  and  Belashek,  1958). 

The  availability  of  the  computer  led  to  advances  in  speech  recognition 
in  the  early  1960s.  Time  normalization  techniques  compensated  for 
variability  in  word  duration.  Small,  portable,  inexpensive  recognizers 
were  developed  that  could  accommodate  a  limited  vocabulary  size.  Various 
strategies  for  enhancing  recognition  accuracy  were  attempted,  based  on 
principles  of  linguistics  and  auditory  perception,  as  well  as  acoustic 
pattern  recognition. 

By  1968,  a  system  was  developed  that  was  capable  of  well  over  90% 
accuracy,  with  a  54  word  vocabulary  (Bobrow  and  Klatt,  1968).  A  few  years 
later,  the  first  commercial  speech  recognizers  appeared  on  the  market. 

A  significant  impetus  was  given  to  speech  recognition  research  in  1972 
when  the  Advanced  Research  Projects  Agency  (ARPA)  sponsored  a  five  year 
program  called  the  Speech  Understanding  Research  (SUR)  project.  The  ARPA 
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SUR  project  funded  five  system  builders  to  pursue  the  objective  of 
developing  the  capability  to  understand  connected  speech  from  many  speakers 
with  a  vocabulary  of  1000  words.  Klatt  (1980)  and  other  chapters  in  Lea 
(1980)  provide  a  comprehensive  review  of  the  advances  arising  from  the  ARPA 
SUR  project. 

The  late  1970's  seemed  to  be  characteri zed  by  continuing  the 
advancements  in  connected  speech  recognition  begun  during  the  ARPA  SUR 
project,  a  prol  iteration  of  commercial  recognizers  of  more  modest 
capabilities,  and  the  development  of  prototype  systems  applications.  In 
1978  the  first  connected  speech  recognizer  became  available  commercially. 

The  present  project  is  based  on  the  prototype  training  systems 
applications  of  speech  recognition  technology.  These  programs,  sponsored 
by  NAVTRAEQU  I  PCEN,  were  begun  in  1973  with  a  study  of  the  feasibility  of 
using  AST  in  an  automated  training  system  for  ground  controlled  approach 
controllers  (Fuege,  Charles,  and  Miller,  1974). 

AST  TERMINOLOGY 

Like  any  new  field  of  endeavor,  speech  technology  has  a  new  set  of 
terms,  or,  more  specifically,  new  connotations  for  old  terms.  The  field  of 
AST  is  an  interdisciplinary  one,  being  derived  largely  from  acoustics, 
linguistics,  and  computing.  The  psychologist  (human  factors  engineer)  also 
is  involved  because  of  his  focus  on  human-computer  interaction,  and  the 
human  element  in  systems  design.  The  terminology  in  AST  stems  from  all  of 
these  roots. 

Recognition  Versus  Generation.  Speech  recognition  and  generation  are  the 
complementary  functions  of  AST.  Speech  generation  also  is  referred  to  as 
synthesis.  The  two  major  techniques  for  producing  speech  by  machine  are 
(1)  to  record,  digitize  and  playback  an  actual  human  voice,  or  (2)  to 
concatenate  and  synthesize  speech  from  a  set  of  machine-generated  phonemes 
or  words.  Digitized  speech  is  very  "normal"  sounding,  but  it  requires  that 
each  word  or  phrase  be  pre-recorded.  Speech  synthesizers,  on  the  other 
hand,  tend  to  sound  unnatural  but  require  no  pre-recordi ng.  Instead  of 
pre-recording,  a  short  program  is  written  to  select  sequences  of  sounds 
that  approximate  the  desired  words.  Summaries  of  speech  generation 
technology  have  given  recently  by  Kaplan  (1980)  and  Michael  is  and  Wiggins 
(1981). 

Speech  recognition  and  speech  generation  allow  the  communication  to 
flow  both  directions  across  the  man-computer  interface.  The  Navy's 
prototype  training  systems  (PARTS  and  ACE)  probably  will  be  forerunners  of 
a  plethora  of  man-computer  systems  in  the  next  decade  that  are  designed 
around  a  speech  interface. 

Computer  speech  recognition  is  enormously  more  difficult  than  speech 
generation.  Therefore,  a  heavy  emphasis  in  this  report  will  be  placed  on 
recognition.  Speech  generation  will  be  given  relatively  short  shift. 
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Recognition  Versus  Understanding.  The  term  "recognition"  often  is  used  in 
the  AST  community  to  refer  to  the  output  of  a  commercial  speech  recognizer. 
This  output  is  based,  in  virtually  all  present  systems,  on  principles  of 
acoustic  pattern  matching.  The  term  "understanding"  often  is  used  to  refer 
to  supporting  software  that  assists  the  recognizer  by  processing 
"knowledge"  about  syntactic,  semantic  and  pragmatic  information  (see  Lea, 
1980).  The  "understanding"  software  enhances  recognition  accuracy  by 
adjusting  the  probabilities  of  vocabulary  items  as  a  function  of  the 
syntactical  rules  of  the  language  and  "knowledge"  of  the  task  at  hand.  The 

understandi ng  software  can  be  considered  a  special  case  of  artifical 

intelligence  (AI)  that  uses  all  available  sources  of  information  to  aid  in 
correctly  understanding  the  intended  meaning. 

Sometimes  the  term  "understanding"  software  is  used  to  imply  the 
initial  selection  of  automated  responses  to  the  recognized  input.  This  use 
of  the  term  strongly  supports  the  contention  that  the  understanding 

software  is  a  type  of  AI.  The  response  selection  function  of 

"understanding"  software  makes  it  analogous  to  human  decision  making  and 
cognition,  as  studied  by  psychologists  and  modeled  by  AI  specialists  (see 
Barr  and  Fiegenbaum,  1979). 

Isolated  Word  Versus  Connected  Speech.  Isolated  word  recognition  (IWP' 
refers  to  recognition  systems  that  are  constrained  to  deal  with  vocabulr  y 
items  separated  by  pauses.  This  type  of  speech  technology  is  also  known  is 
isolated  phrase  recognition  (IPR),  since  any  string  of  uninterrupted 
acoustic  input  less  then  some  maximum  time  (typically  1  to  3  seconds)  is 
processed  as  a  single  unit.  The  unit  may  consist  of  a  single  word,  such  as 
continue,"  several  words,  such  as  "turn  right"  or  a  series  of  words,  such 
as  "this  is  your  final  controller  how  do  you  hear  me".  Each  of  these  could 
bo  defined  for  the  IWR  system  as  a  single  unit  of  speech,  i.e.,  an 
utterance. 

Recognition  systems  with  the  capability  to  separate  strings  of  inrut 
are  called  connected  (or  limited  continuous)  speech  recognizers  (CSR). 
These  systems  must  segment  the  input  string  into  its  constituent  units  as 
well  as  recognize  them.  A  considerable  amount  of  higher  level  processing 
often  is  necessary  to  parse  the  input  correctly.  The  most  successful 
system  from  the  ARPA  SUR  project,  HARPY,  for  example,  used  syntactic 
knowledge,  lexical  knowledge,  juncture  rules,  and  phonemic  knowledge  to 
achieve  good  (>90%)  recognition  accuracy  of  connected  speech  (Lowerre  and 
Reddy,  1980).  It  should  be  noted  that  for  many  practical  applications 
requiring  real-time  speech  interaction,  large  processing  capacities  are 
necessary  to  avoid  the  time  constraints  imposed  by  integrating  these 
knowledge  sources.  Some  of  the  difficulties  associated  with  connected  or 
continuous  speech  recognition  were  discussed  by  Levinson  and  Liberman 
(1981),  who  have  been  developing  these  techniques  at  Bell  Labs  for  large 
and  relatively  unconstrained  vocabularies. 

For  IWR  systems  the  capacity  requirements  are  less,  but  the  user  must 
separate  the  input  string  by  pausing  before  and  after  each  utterance.  The 
implications  of  the  pause  requirements  depend  on  the  task.  Where  a 
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relatively  naive  user  must  input  several  strings  of  words/phrases  within  a 
perceived  time  limit,  pauses  are  apt  to  be  omitted,  causing  a  reduction  in 
recognition  accuracy.  The  PARTS  prototype  trainer  suffered  some  reduction 
in  accuracy  due  to  inexperienced  users  (precision  approach  radar  trainees) 
failing  to  pause  adequately  between  phrases  (McCauley  and  Semple,  1980). 
The  effect  is  likely  to  be  most  pronounced  when  the  user  tries  to  hurry 
because  of  task-related  time  requirements.  However,  recent  advancements  in 
the  technology  reportedly  have  reduced  the  pause  time  to  less  than  50 
milliseconds.  This  capability  diffuses  the  distinction  between  IWR  and 
limited  connected  speech  recognition. 

The  difficulties  associated  with  machine  segmentation  of  a  digit 
string  into  individual  digits  were  discussed  by  Flanagan,  Levinson, 
Rabiner,  and  Rosenberg  (1980).  In  some  cases  digit  (word)  boundaries  may 
not  exist  because  of  coarticulation.  For  example,  in  the  digit  string 
"199,"  the  final  "n"  in  one  becomes  the  initial  "n“  in  nine,  leaving  no 
clear  acoustical  evidence  for  automatic  segmentation  into  three  digits. 

Speaker  Dependence  Versus  Independence.  Most  recognition  systems  today  are 
speaker  dependent.  They  sample  the  speech  of  the  individual  user  to  create 
a  reference  pattern,  or  template,  for  each  vocabulary  item.  The  number  of 
samples  for  each  vocabulary  item  varies  from  two  to  ten,  depending  on  the 
manufacturer.  This  speech  sampling  process  has  been  called  various  names, 
such  as  voice  data  collection  (VDC),  voice  training,  and  enrollment.  No 
matter  what  it  is  called,  the  process  takes  time.  An  extreme  example  would 
be  a  system  which  required  10  samples  per  vocabulary  item.  A  moderate 
vocabulary  size,  such  as  100  words/ phrases ,  would  require  each  user  to 
produce  1000  speech  samples  before  the  system  would  be  useable.  Once  a 
sample  is  taken,  however,  it  may  be  valid  for  extended  periods  of  time  on 
some  systems  with  experienced  users.  Poock  (1981)  reported  good 
recognition  accuracy  with  speech  reference  patterns  sampled  two  years 
previously. 

Speaker  independent  systems  require  no  individualized  speech  reference 
patterns.  Ideally,  any  person  who  speaks  the  language  can  use  the  system 
without  prior  speech  samples.  This  type  of  system  eliminates  the  time 
requirement  for  speech  sampling,  but  it  must  be  able  to  accomnodate  both 
the  inter-  and  intra-speaker  variabilities.  This  additional  source  of 
variability  must  be  finessed  by  the  use  of  "understanding"  software,  such 
as  subsetting  the  vocabulary  according  to  syntactical  rules  of  the 
t.ask/appl  ication  (see  Flanagan,  et  al.,  1980,  for  an  example). 

HUMAN  FACTORS  ISSUES  IN  AST  TRAINING  SYSTEMS 

High  Expectations.  Interacting  with  an  automated  system  by  voice  is 
currently  a  novel  experience  for  most  people.  Naive  users  are  faced  with 
uncertainties  about  what  is  expected  of  them,  and  what  they  should  expect 
of  the  system.  Although  most  manufacturers  report  99%  recognition 
accuracy,  that  figure  usually  is  obtained  under  laboratory  conditions  by 
experienced  speakers.  A  potential  problem  for  AST  acceptance  is  overly 
high  expectations  (Van  Hemel ,  Van  Hemel ,  King  and  Breaux,  1980).  Despite 
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the  impression  given  by  science  fiction  movies,  speech  interactive  systems 
presently  are  far  from  approaching  the  richness  and  complexity  of  human 
dialog. 

Speech  Sampling.  IWR  systems  require  the  user  to  repeat  each  phrase  from  2 
to  10  times.  In  an  automated  training  system,  this  procedure  can  be 
interleaved  with  instructional  material  to  prevent  tedious  repetition 
(Hicklin,  Barber,  Bol lenbacker,  Grady,  Harry,  Meyn,  and  Slemon,  1980). 
However,  at  least  one  key  scientist  advocates  successive  sampling  on  each 
vocabulary  item  (Poock,  1981,  personal  communication).  Poock  suggests  that 
phrase  repetitions  allow  the  user  to  introduce  some  variability  into  the 
sample,  creating  a  reference  pattern  that  is  more  "forgiving"  of  subsequent 
speech  variability. 

No  matter  what  sampling  procedure  is  used,  trainees  may  find  the 
process  boring  after  the  first  few  repetitions,  as  reported  in  the  PARTS 
evaluation  (McCauley  and  Semple,  1980). 

A  related  issue  is  how  to  elicit  speech  samples.  Visual  prompts 
displayed  on  a  CRT  seem  to  be  the  most  common  method.  Speech  synthesis 
also  can  be  used  as  a  prompt,  but  this  procedure  sometimes  induces 
unintentional  mimicking  of  the  synthesized  speech  characteristics  (McCauley 
and  Semple,  1980). 

Context  effects  often  are  advocated  for  producing  good  speech  samples. 
Context  implies  both  physical  and  task  context.  For  example,  if  the 
Precision  Approach  Radar  trainee  produces  speech  samples  by  reading  from  a 
list  on  a  CRT,  his  speech  pattern  may  be  considerably  different  when 
controlling  a  simulated  final  approach. 

Practice  Effects.  New  users  of  AST  vary  in  their  obtained  recognition 
accuracy.  With  the  exception  of  the  user  who  immediately  achieves  very 
high  accuracy,  practice  and  experience  usually  result  in  better  recognition 
performance.  This  change  probably  occurs  through  increased  voice  control, 
leading  to  more  consistent  speech  rate,  inflection,  and  volume. 

Increased  voice  control  also  is  evidenced  by  the  user's  response  to  a 
recognition  error.  The  normal  reaction  to  a  human  recognition  error  is  to 
speak  more  slowly,  louder,  and  with  more  emphasis.  This  reaction  may  be 
effective  for  the  human  listener,  but  it  is  detrimental  to  automated 
recognition.  The  experienced  AST  user  reacts  to  an  error  by  repeating  the 
phrase  with  aplomb,  in  the  same  manner  he  produced  the  original  speech 
sample. 

Recognition  Test  and  Sample  Updating 

Most  AST  systems  have  a  mode  of  operation  designed  to  test  recognition 
accuracy.  The  user  speaks  any  of  the  trained  vocabulary  phrases,  and  the 
recognized  phrase  is  displayed  on  a  CRT,  or  otherwise  fed-back  to  the  user. 
The  recognition  test  (or  "speech  test")  mode  is  beneficial  because  it 
enables  voice  control  practice,  allows  testing  the  limits  of  variability 
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tolerated  by  the  system,  and  fosters  the  user's  confidence  in  the  system. 
Another  important  function  of  the  recognition  test  mode  is  to  determine 
whether  the  current  set  of  speech  samples  is  sufficient  to  achieve 
acceptable  accuracy.  For  example,  a  fatigued  user  may  suspect  that 
recognition  accuracy  is  declining  during  a  long  session.  He  can  use  the 
test  mode  to  determine  whether  his  suspicions  are  warranted.  He  then  has 
the  option  to  update  the  reference  patterns  of  any  word/phrases.  This 
cycle  of  test-and-update  can  be  used  for  any  reason  that  would  induce 
changes  in  the  voice  relative  to  the  reference  pattern  (i.e.,  cold,  flu, 
fatigue,  smoking,  etc.).  Connolly  (1979)  addressed  this  issue  in  his 
studies  on  voice  data  entry  in  the  Federal  Aviation  Administration. 

Styl ization.  The  speech  stylization  necessary  for  IWR  systems  is  a 
definite  constraint  for  some  applications.  It  puts  the  burden  on  the  user 
to  modify  a  highly  overlearned  behavior  (speech)  to  compensate  for  the  lack 
of  segmentation  capability  of  the  IWR  system.  Careful  definition  of  the 
vocabulary  items  is  essential  to  avoid  unnatural  pauses.  But  even  then, 
some  users  may  be  expected  to  have  difficulty  inserting  pauses  correctly, 
and  occasionally  refraining  from  pausing  between  the  words  of  a  phrase 
unit.  For  example,  in  the  PARTS  vocabulary  the  following  phrase  woul'1  not 
be  recognized  if  a  pause  were  inserted,  "This  is  your  final  controller,  how 
do  you  hear  me?"  (Hick! in,  et  al . ,  1980).  The  trainees  simply  had  to  learn 
to  refrain  from  pausing  after  the  word  "control ler. " 

Speech  stylization  is  not  an  important  limitation  in  tasks  with  short, 
well  defined  command  phrases.  Many  Navy  training  tasks  would  be  in  this 
category,  including  the  GCA  task  trained  by  PARTS  system.  Speech  tasks 
that  involve  lengthy  phrases  without  explicit  rules  for  pauses  are  more 
difficult  for  present  speech  recognition  systems.  Restructuring  the 
vocabulary  might  be  necessary  before  AST  could  be  applied  for  training  such 
a  task. 

Recognition  Feedback.  Feedback  to  the  user  is  desirable  in  any  AST 
application.  It  is  particularly  important  in  systems  where  a  considerable 
probability  of  recognition  error  is  experienced  (i.e.,  greater  than  5  or 
10%).  Feedback  can  be  presented  in  several  ways  -  alphanumeric  readout  on 
CRT,  speech  generation,  or  any  unambiguous  change  in  the  visual  or  audio 
display.  Without  feedback,  the  trainee  in  a  continuous  task,  such  as  an 
air  controller,  has  no  way  to  verify  that  his  verbal  information  was 
recognized  correctly.  This  places  an  additional  burden  on  the  trainee,  to 
assess  the  situation,  hypothesize  whether  his  transmission  was  recognized 
correctly,  then  repeat  it,  modify  it,  or  continue  the  problem. 

The  timing  of  feedback  can  be  an  important  human  factors  issue. 
Normally,  it  should  occur  within  a  second  or  two  from  completion  of  the 
word/phrase.  The  exact  time  constraints  are  dependent  on  the  temporal 
aspects  of  the  task,  especially  the  need  for  feedback  before  continuing  to 
the  next  verbal  transmission. 
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One  difficulty  with  providing  feedback  to  the  user  is  to  avoid 
interfering  with  the  primary  task.  How  to  make  the  recognition  feedback 
available  to  the  trainee  without  being  interruptive  is  an  interesting 
system  design  issue.  The  answer  clearly  depends  on  the  nature  of  the  task 
and  the  changing  demands  on  the  trainees’  attentional  capacities 
(Chatfield,  Klein,  and  Coons,  1981;  Norman  and  Bobrow,  1976). 

VARIABLES  AFFCCTING  ACCURACY 

Speaker  Variables 

A  person's  speech  variability  over  time  is  the  primary  challenge  for 
speaker-dependent  systems.  Speaker- independent  systems  are  faced  with  the 
additional  problem  of  variability  between  speakers.  The  human  factors 
engineer  and  the  systems  designer  should  consider  the  potential  avenues 
open  to  them  for  reducing  the  sources  of  variability  that  detract  from  AST 
system  performance. 

Between- speaker  variability  is  largely  beyond  the  control  of  the 
systems  designer.  Pre-testing  or  otherwise  selecting  the  user  population 
can  be  one  tool  for  reducing  between-speaker  variability.  Extensive  speech 
training  is  another  approach,  but  not  a  practical  one  for  most 
appl ications.  Some  variables  associated  with  between-speaker  variability 
are: 

o  Sex  (fundamental  pitch) 
o  Speech  rate 

o  Stress  patterns  (prosodies) 
o  Dialect 


Speech  recognizers  have  traditionally  performed  better  with  male 
speakers  than  female.  According  to  Welch  (1981,  personal  communication), 
this  difference  is  greater  in  connected  speech  recognition  systems  than  in 
IWR  systems.  Females,  however,  have  obtained  99%  recognition  accuracy  in  a 
number  of  laboratory  tests  of  recognizers  (Doddington  and  Schalk,  1980),  so 
it  should  not  be  concluded  the  AST  is  inappropriate  for  female  users. 

All  of  the  characteristics  of  an  individual's  speech  that  make  it 
unique,  such  as  pitch,  inflection,  and  stress  patterns,  make  it  difficult 
to  design  a  speaker- independent  recognition  system.  It  must  be  immune  to 
those  individual  differences,  and  be  capable  of  recovering  (or 
constructing)  the  invariants  in  the  speech  that  correspond  to  the  message. 

Speaker-dependent  systems  attempt  to  account  for  intra -speaker 
variability  through  speech  sampling,  followed  by  acoustic  pattern  matching. 
One  of  t.he  difficulties,  however,  is  for  the  system  to  be  flexible  enough 
to  accommodate  a  range  of  variability  within  the  individual's  speech,  while 
maintaining  an  acceptably  low  proportion  of  false  positive  recognitions. 
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The  variability  (or  lack  of  consistency)  in  the  individual's  speech, 
therefore,  becomes  an  important  factor  that  affects  recognition  accuracy  in 
speaker-dependent  systems. 

Some  of  the  myriad  of  variables  that  can  affect  an  individuals 
consistency  of  speech  are: 

o  fatigue 

o  evaluation  stress 
o  time  stress 
o  other  task  stress 
o  health/medical 
o  medication  and  drugs 
o  affective  state 
o  general  activity  level 
o  attitude/motivation 
o  practice 


Most  current  recognizers  may  be  considered  susceptible  to  errors  when 
the  consistency  of  speech  changes  due  to  any  of  these  factors.  Often  the 
human  listener  is  unable  to  detect  subtle  differences  in  the  way  a  speaker 
says  a  phrase  to  the  system,  but  the  differences  will  be  sufficient  to 
change  the  outcome  from  a  correct  to  an  incorrect  recognition. 

Personal  stress  and  tension  may  play  an  important  role  in  recognition 
accuracy.  Stress  often  will  result  in  a  pitch  shift,  an  increased  speech 
rate,  and  a  tendency  to  omit  pauses.  These  stress  effects  may  ho 
particularly  important  for  training  applications,  where  the  trainee  may  be 
subject  to  a  number  of  stressors,  notably  evaluation  stress  and  time 
stress.  The  psychological  aspects  of  time  stress  during  training  are 
discussed  by  Chatfield  et  al . ,  (1981)  who  suggest  that  the  time  for 
cognitive  processing  is  taxed  during  training. 

Stress  effects  could  be  part  of  the  explanation  for  the  discrepancy 
between  the  mean  recognition  accuracy  results  of  nearly  98%  reported  by 
Poock  (1981)  and  76%  in  PARTS  (McCauley  and  Semple,  1980).  Both  these 
systems  used  essentially  the  same  model  of  recognizer.  Poock 's  graduate 
students  were  experienced  in  the  task  (a  conmand  and  control  computer  net) 
and  they  were  under  no  particular  evaluation  stress.  The  trainees  in  the 
PARTS  evaluation  were  younger,  learning  the  precision  approach  radar  task, 
responding  to  an  event-driven  (time-critical)  situation,  and  being 
evaluated  after  each  trial.  Each  of  these  factors  could  be  associated  with 
increased  stress.  Poock  (1981)  noted  that  the  average  error  rate  increased 
in  week  8  (of  his  20  week  study),  which  coincided  with  the  school’s 
examination  schedule.  He  suggested  that,  "it  is. ..a  small  indication  of 
possible  stress  factors  at  work.  It  helps  to  point  out  that  much  remains 
to  be  done  in  the  whole  area  of  environmental  and  psychological  stress 
effects  on  voice  recognition  performance"  (Poock,  1981,  p.  9). 
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Experienced  speech  system  users  commonly  report  99%  accuracy  with  a 
variety  of  systems  (see  Doddington  and  Schalk,  1980).  The  reason  for  the 
lower  accuracy  rates  in  many  practical  applications  needs  to  be 
investigated.  Slight  improvements  in  recognition  accuracy  can  mean  large 
gains  in  system  performance  when  an  entire  real-time,  interactive 
simulation  (and  subsequent  performance  measurement)  is  being  driven  by  the 
recognized  speech  input. 

Environmental  Variabilites 

Environmental  variabilities  as  well  as  speaker  variabi  1  i ties  can 
contribute  to  recognition  accuracy.  Potential  environmental  factors  are: 

o  Ambient  Noise 
o  Vibration 

o  Sustained  Acceleration 


Ambient  noise  can  have  a  significant  effect  on  accuracy,  particularly 
when  it  varies  between  speech  sampling  and  subsequent  attempts  at 

recognition.  Coler  (1980)  has  reported  the  detrimental  effects  of 

helicopter  noise  on  recognition.  But  the  use  of  a  close-talking, 
noise-cancel  1 ing  microphone  and  obtaining  speech  samples  in  the  noisy 
environment  can  result  in  relatively  good  performance  (Coler,  Huff, 
Plummer,  and  Hitchcock,  1978;  Coler,  in  press).  The  effects  of  noise  on 
performance  must  be  determined  for  each  case  because  the  interfering 

effects  of  the  noise  are  a  complex  function  of  the  overlap  and  temporal 
relationship  between  the  noise  spectrum  and  the  speech  spectrum. 

The  effects  of  vibration  and  sustained  acceleration  on  speech 

recognition  accuracy  have  not  been  studied  systematical ly.  Plans  have  been 
established  to  investigate  acceleration  (g  load)  effects  on  the  centrifuge 
at  the  Naval  Air  Development  Center,  Warminster  (Harris,  in  press). 
Vibration  effects  are  certain  to  influence  recognition  accuracy  within 
certain  frequency  and  amplitude  domains,  but  the  authors  are  unaware  of  any 
studies  on  this  issue. 

Wind  noise  is  another  environmental  variable  that  can  potentially 
affect  recognition  performance.  Wind  protective  devices  are  made  for  many 
microphones,  but,  again,  the  effects  of  wind  and  protection  devices  on 
computer  speech  recognition  have  not  been  systematically  investigated. 

In  general,  environmental  factors  can  influence  speech  recognition  at 
several  points:  within  the  human  speech  apparatus; at  the  microphone;  and  at 
the  computer.  Physical  disturbance  of  the  speaker,  such  as  motion, 
vibration,  or  sustained  acceleration,  can  produce  a  direct  mechanical 
influence  on  his  organs  of  speech  production.  Indirect  effects  can  arise 
from  environmental  factors  such  as  low  humidity  and  dust  that  affect  the 
vocal  chords.  In  extreme  temperatures,  such  as  found  in  high  altitude 
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flight,  the  effects  might  be  most  critical  on  the  computer  itself. 
Ruggedized  components  would  be  necessary  to  support  automated  speech 
technology  in  such  hostile  environments. 

Vocabul ary.  Definition  of  the  task  vocabulary  is  an  important  part  of 
achieving  good  recognition  accuracy.  In  the  general  case,  vocabulary  size 
is  related  to  accuracy.  This  is  an  intuitively  appealing  concept,  because 
the  probability  of  erroneously  recognizing  an  utterance  increases  as  the 
number  of  possible  utterances  increases.  However,  intuitively  appealing 
concepts  are  not  always  correct,  and  at  least  one  study  has  reported  no 
significant  relationship  between  vocabulary  size  and  error  rate.  Poock 
(1981),  using  an  IWR  system,  found  no  change  in  recognition  error  rate  as 
the  vocabulary  size  increased  from  20  to  240  words. 

The  number  of  vocabulary  items  may  be  less  important  than  the 
confusibil ity  of  the  items.  Probably  the  worst  case  for  automated 
recognition  is  a  vocabulary  containing  many  single-syllable  words.  In  the 
ACE  system,  for  example,  the  words  "Port"  and  "Four"  are  frequently 
confused  (McCauley,  Root,  and  Muckier,  in  press).  Smith  and  Samber  (1980) 
described  the  problem  succinctly. 

As  more  words  are  added  to  the  vocabulary,  the  more 
likely  it  is  that  words  will  be  confused  with  one 
another.  Sometimes  this  is  because  one  word  is  a 
subpart  of  another,  such  as  "Plea"  being  confused  with 
"Please."  Other  times  the  confusion  comes  because  words 
have  similar  acoustic  descr i pt i ons . . . [and  the] 
processors  cannot  distinguish  between  the  acoustic 
patterns.  An  example  of  this  is  the  words  "What"  and 
"Watt"  (Smith  and  Samber,  1980,  p.  151). 


Vocabulary  size  and  confusabi 1 ity  must  be  weighed  against  the  larger 
picture  of  the  particular  task  or  application.  Imagine  an  hypothetical 
system  where  operational  requirements  would  allow  either  "Nine"  or  “Niner" 
to  be  transmitted,  i.e.,  it  was  not  the  case  that  only  one  of  two  was 
correct.  The  speech  system  designer  could  put  both  words  in  the 
vocabulary,  and  both  would  be  recognized  as  "Nine."  In  this  case,  he  did 
two  things  that  are  usually  not  good,  but  in  this  case,  they  improved  the 
system:  1)  He  increased  the  vocabulary  size,  and  2)  he  allowed  two  words 
with  potentially  high  confus abi 1 ity  (because  one  is  subset  of  the  other). 
However,  since  both  words  are  "mapped"  to  the  same  recognized  output,  the 
system  is  more  flexible  and  "friendlier"  to  the  user.  It  allows  either 
word  to  be  spoken,  and,  even  if  a  confusion  error  is  made,  the  final 
understanding  is  the  same. 

In  summary,  several  vocabulary  factors  can  affect  recognition 
accuracy.  These  include  the  total  vocabulary  size,  the  number  of  short 
(one- syllable)  words,  and  their  acoustic  similarity. 
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POTENTIAL  APPLICATIONS 

The  potential  applications  of  automated  speech  recognition  are 
numerous,  almost  as  unlimited  as  the  use  of  speech  itself.  The  commercial 
and  military  applications  were  outlined  by  Lea  (1980),  as  follows: 

Commercial  Applications 

o  Package  sorting 
o  Quality  control  and  inspection 
o  Programming  numerical ly-control led  machines 
o  Voice-actuated  wheelchair 
o  Banking  transactions 
o  Security  and  accesss  control 

Military  Applications 
o  Cartography 

o  Training  air  traffic  controllers 
o  Cockpit  control  actuation 
o  Spotting  keywords  in  monitored  conversations 
o  Command  and  control 


In  another  analysis  of  the  AST  marketplace,  Nye  (1980)  noted  that  "the 
need  is  clear  and  the  future  bright  for  'humanized'  man/machine 
communicat ions. "  He  cited  several  types  of  present  and  potential 
applications  of  AST,  including: 

o  Data  entry 

o  Education  and  training 
o  Medical 
o  Inspection 
o  Banking 
o  Accounting 
o  Computer  I/O 
o  Source  Entry 
o  Machine  tool 
o  Qual ity  control 
o  Material  hand! ing 


Present  limitations  on  AST  applicability  stem  from  several  sources 
including  cost,  accuracy,  speech  stylization  requirements,  speech  data 
collection,  and  vocabulary  constraints.  Despite  some  limitations,  present 
recognition  systems  are  sufficient  to  support  many  applications,  such  as 
command  and  control  (Poock,  1980),  data  entry  (Connolly,  1979),  real-time 
computer  interaction/simulation  (Breaux  and  Goldstein,  1975;  Grady  and 
Hicklin,  1976,  Halley,  King,  and  Regelson,  1979),  and  non  real-time 
interaction,  such  as  Intelligent  Computer  Assisted  Instruction  (ICAI) 
(Meisel,  in  press).  Many  commercial  uses  of  speech  recognition  are 
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possible,  but  information  is  available  on  only  a  few  early  attempts  by 
organizations  such  as  the  U.S.  Postal  Service  (Craft,  1981)  and  Lockheed 
(Lerman  1980). 

The  first  commercial  recognizer  appeared  in  1972.  There  currently  are 
at  least  eight  manufacturers  of  conmercial  speech  recognizers  (Nye,  1981). 
Their  products  range  in  price  from  less  than  $1,000  to  nearly  $100, 000. 
The  number  of  recognizers  on  the  market  has  been  increasing  in  the  past  few 
years.  The  price  range  also  seems  to  be  increasing  because  some 
manufacturers  are  seeking  low  cost,  limited  capability  machines  for  the 
hobbyist,  while  others  are  reaching  for  the  sophisticated  realm  of 
connected  or  continuous  speech  with  large  vocabularies. 

Current  advances  in  electronic  chips  are  likely  to  reduce  the  cost 
while  increasing  the  capability  of  available  speech  recognizers.  The 
number  of  military  and  commercial  applications  can  be  expected  to  grow 
rapidly  during  the  1980s.  Continuous  speech  understand! ng  of  large 
vocabularies  may  require  years  of  research  on  syntactic  and  semantic 
processing,  but  automatic  recognition  of  discrete  words  and  phrases,  with 
vocabulary  sizes  up  to  1000  units,  is  a  technology  of  today. 
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SECTION  IV 

REVIEW  OF  PRIOR  NAVY  AST  TRAINING  SYSTEMS 


The  Navy  has  sponsored  the  development  of  two  experimental  prototype 
training  systems  centered  around  automated  speech  recognition  and 
generation.  A  third  system  is  in  the  functional  design  stage.  These 
prototype  systems  have  contributed  a  great  deal  to  the  knowledge  of  the 
capabilities  and  limitations  of  current  speech  technology  in  support  of 
automated  training.  The  lessons  learned  from  these  prototype  developments 
enable  the  potential  of  AST  training  systems  to  be  assessed.  Because  these 
two  systems  represent  the  state-of-the-art,  as  of  1980-81,  a  brief  review 
of  each  will  be  given.  Examples  from  these  systems  then  will  be  used 
repeatedly  throughout  this  report. 

PRECISION  APPROACH  RADAR  TRAINING  SYSTEM  (PARTS) 

Two  names  have  been  used  to  refer  to  the  first  application  of  AST  to  a 
real-world  training  problem  -  PARTS,  and  Ground  Controlled  Approach 
Controller  Training  System  (GCA-CTS).  This  training  system  was  developed 
by  Logicon,  Inc.  under  the  sponsorship  of  NAVTRAEQUIPCEN  during  the  time 
period  of  1974  to  1979  (Fuege,  et  al.,  1974;  Breaux,  1976;  Grady  and 
Hicklin,  1976;  Grady,  Hick! in  and  Porter,  1980;  Hicklin,  Nowell  and 
Petersen,  1978;  Hicklin,  et  al . ,  1979a,  Hicklin,  et  al . ,  1979b,  Hicklin,  et 
al.,  1980).  An  independent  evaluation  of  PARTS  was  conducted  by  Canyon 
Research  Group,  Inc.  at  the  Navy  Air  Traffic  Control  School  (McCauley  and 
Semple,  1980). 

The  speech  recognition  technology  incorporated  in  PARTS  was  a  an  IWR 
system,  the  Threshold  500,  manufactured  by  Threshold  Technology,  Inc.  and 
supported  by  "understanding"  software  developed  by  Logicon.  A  vocabulary 
of  107  phrases  was  used,  varying  in  complexity  from  single  digits  to  "This 
is  your  final  controller  how  do  you  hear  me?" 

The  speech  generation  technology  included  both  digitized  playback  and 
a  speech  synthesizer  by  Votrax  (Model  VS  6.4). 

The  PARTS  was  a  complex,  sophisticated  prototype  training  system.  It 
included  not  just  the  real-time  speech-interactive  simulation  of  a  PAR 
approach,  but  other  advanced  technologies  such  as  automated  instruction, 
automated  performance  measurement/evaluation,  and  automated  syllabus 
control.  A  schematic  diagram  is  shown  in  Figure  2.  Detailed  descriptions 
of  the  system  can  be  found  in  Barber  and  Hicklin  (1980)  and  Hicklin,  et 
al.  (1980). 

In  the  future,  it  is  likely  that  the  PARTS/GCA-CTS  development  will  be 
thought  of  as  a  pioneering  effort  that  provided  impetus  toward  the 
practical  application  of  advanced,  automated  technology  in  support  of 
instruction  and  training.  It  was  not  without  shortcomings,  however,  as 
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Figure  2.  PARTS  Training  System  Diagram  (from  Logicon,  1979) 
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might  be  expected  in  a  prototype  development  of  this  magnitude.  The 
evaluation  of  PARTS  concluded  that  computer  speech  recognition  is... 

sufficiently  advanced  to  begin  applying  it  in 
appropriate  training  tasks...  From  a  training 
effectiveness  perspective,  the  major  problem  encountered 
in  PARTS  speech  recognition  was  the  occurrence  of  a 
critical  recognition  error  that  sometimes  caused  loss  of 
control  of  the  [PAR]  approach...  Revision  of  the 
software  supporting  PARTS  speech  recognition  would 
eliminate  the  problem  of  critical  recognition  errors. 

The  [problem]  serves  to  emphasize  the  importance  of 
developing  task-specific  [understanding]  software  for 
each  application  of  automated  speech  technology 
(McCauley  and  Semple,  1980,  p.  119). 

The  evaluation  recommended  that  the  feasibility  of  applying  AST  to 
training  should  be  studied  on  a  wider  basis  for  Navy  air  traffic 
controllers,  including  area  surveillance  radar  (ASR),  radar  air  traffic 

control  facility  (RATCF)  and  carrier  air  traffic  control  center  (CATCC). 

The  difficulties  encountered  in  PARTS  speech  recognition  can  be 

sumnarized  by  the  average  transmission  recognition  accuracy  (TRA),  which 
was  76%  for  the  22  students  who  completed  the  final  performance  test.  The 
TRA  measure  underestimates  word/phrase  recognition  accuracy  because  some 
transmissions  consist  of  several  words/ phrases,  such  as  "Turn  right  heading 
160,  over."  This  transmission  consisted  of  five  items  in  the  vocabulary, 
and  would  be  said  "Turn  right  heading  PAUSE  one  PAUSE  six  PAUSE  zero  PAUSE 
over."  A  recognition  error  in  one  or  more  of  the  five  items  was  scored  as 
a  transmission  error.  This  measure  might  be  judged  as  harsh  by  AST 

manufacturers ,  but  in  the  PARTS  task,  the  aircraft  would  not  respond 

appropriately  unless  the  entire  transmission  was  "understood". 

The  difficulties  arising  from  this  mediocre  recognition  performance 
can  be  understood  by  reference  to  the  previous  figure  showing  the  schematic 
diagram  of  PARTS.  Speech  recognition  provides  the  input  to  the  simulation 
(aircraft/pilot/*environment  model),  performance  measurement,  and  subse¬ 
quently,  evaluation  and  recordkeeping,  feedback,  and  syllabus  control.  Any 
errors  in  speech  recognition,  therefore,  cascaded  throughout  the  remaining 
system  functions.  This  is  a  knotty  system  design  problem;  one  which  will 
be  discussed  in  more  detail  later  in  this  report.  Suffice  to  say  that  any 
improvements  in  speech  recognition  accuracy  would  have  resulted  in 
substantial  gains  in  PARTS  training  effectiveness.  Further  development  of 
syntactical  and  other  "understanding"  software  may  have  produced  such 
improvements  (see  Strieb  and  Dow,  1980). 

Finally,  it  should  be  mentioned  that  the  PARTS  speech  recognition 
system  was  dealing  with  a  difficult  situation  because  the  users  were 
students  who  were:  1)  under  learning  stress;  2)  under  evaluation  stress; 
3)  inexperienced  in  radio  communication;  and  4)  only  on  the  system  for  a 
short  time  (4  1/2  days).  A  recognition/understanding  system  more  tolerant 
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of  users  who  are  inexperienced  and  stressed  would  produce  an  immediate  leap 
in  overall  system  effectiveness.  Even  using  the  original  recognition 
system,  increased  recognition  accuracy  could  be  achieved  through  careful 
human  factors  integration  of  the  speech  capability  into  the  overall  system. 

The  effectiveness  of  the  automated  instruction,  performance 
measurement,  and  syllabus  control  functions  of  PARTS  was  reduced  by  both 
speech  recognition  errors,  as  described  above,  and  by  design  deficiencies, 
some  of  which  may  be  unavoidable  in  prototype  systems.  These  issues  have 
been  discussed  by  McCauley  and  Semple  (1980)  and  will  not  be  reiterated 
here.  However,  many  of  the  lessons  learned  from  the  PARTS  prototype  are 
reflected  in  the  design  guides  herein. 

AIR  CONTROLLER  EXERCISER  (ACE) 

The  ACE  system  has  a  lineage  similar  to  PARTS.  It  has  been  known  by 
another  name,  the  Prototype  Automated  Controller  Training  System  for  Air 
Intercept  Controllers  (PACTS-AIC).  It  was  developed  by  Logicon,  inc.  under 
sponsorship  of  NAVTRAEQUIPCEN  during  the  time  period  of  1977  to  1981 
(Anders,  Grady,  Nowell,  and  Overton,  1979;  Clark,  Halley,  Regel  son,  Slemon, 
Van  Steeg,  et  al . ,  1979;  Grady,  Hicklin,  and  Miller,  1977;  Grady,  Porter, 
Satzer,  and  Sprouse,  1979;  Halley,  Hooks,  Lankford,  and  Nowell,  1979; 
Halley,  King,  and  Regelson,  1979;  and  Smith,  Granberry,  Halley,  Regelsen, 
and  King,  1980).  An  independent  evaluation  of  ACE  is  underway  in  1981/82 
at  the  Navy  Fleet  Combat  Training  Center  Pacific. 

ACE  uses  the  most  sophisticated  connected-speech  recognition  (CSR) 
technology  currently  on  the  market,  the  NEC  DP-100,  manufactured  by  the 
Nippon  Electric  Company.  Supporting  software  was  developed  by  Logicon. 
The  speech  generation  aspects  of  ACE  include  both  computer  synthesized 
speech  (by  VOTRAX)  and  digitized  speech  recording  and  playback. 

ACE  was  designed  to  use  the  connected  speech  capability  for  digit 
strings  in  the  AIC's  vocabulary,  such  as  giving  aircraft  headings,  or 
bearing  and  range  to  a  "bogey"  (potential  enemy)  aircraft.  For  example, 
"Bogey  two  seven  zero,  twenty  three"  tells  the  pilot  that  the  location  of 
the  bogey  is  bearing  270°  at  23  miles.  Updated  bearings  and  ranges  are 
given  so  frequently  in  this  task  that  the  requirement  to  pause  between  each 
digit  would  detract  from  the  realistic  pacing  of  the  task.  The  CSR 
technology  is  better  suited  than  IWR  for  the  fast  pace  of  the  digit  strings 
spoken  by  the  AIC  trainee.  Other  portions  of  the  AIC's  vocabulary  tend  to 
consist  of  short  phrases,  compatible  with  the  pauses  required  by  IWR 
technology.  The  connected  speech  capability  of  the  NEC  system  allows 
concatenation  of  up  to  five  words/phrases  within  a  five-second  interval. 

The  speech  generation  subsystem  in  ACE  includes  several  current 
methods.  Computer  synthesized  speech  is  used  to  simulate  the  messages 
from  the  pilot.  Digitized  speech  is  played  back  to  simulate  messages  from 
a  tactical  controller,  and  the  trainee's  speech  is  recorded  and  digitized 
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to  enable  a  training  scenario  to  be  replayed.  The  remaining  audio 
capability  is  part  of  a  video  disc  system,  which  includes  both  audio  and 
video. 

Although  speech  technology  is  the  central  research  issue  in  ACE, 
several  other  emerging  technologies  also  are  included.  Video  disc  is  used 
for  presenting  instructional  information  and  demonstrations.  It  provides 
rapid  access  to  audio-visual  materials  and  can  display  either  still  or 
moving  video.  Full  simulation  of  the  AIC  operating  environment  is  achieved 
through  the  use  of  pilot/aircraft  models  that  interact  in  real-time  with 
the  trainee's  verbal  transmissions.  The  trainee's  console  is  a  simulated 
Navy  Tactical  Data  System  (NTDS)  console  with  extra  features,  such  as  a 
CRT,  to  enhance  training.  Automated  adaptive  training  is  achieved  by 
criteron-based  assessment  of  trainee  proficiency.  Instructional  sequence 
decisions  range  from  remediation  through  rapid  advancement  by  optional 
"challenge"  opportunities.  Records  of  trainee  performance  are  kept  by 
automated  data-base  management.  Performance  summaries  are  available  for 
perusal  by  the  instructor.  Automation  of  the  entire  job  of  the  human 
"pseudo-pilot"  is  enabled  by  the  combination  of  computer  speech 
recognition,  synthesis,  and  pilot/aircraft  modeling.  Further  reduction  in 
the  workload  of  instructional  personnel  is  achieved  by  the  automation  of 
many  of  the  tasks  and  functions  of  the  instructor.  Automated  instruction, 
task  simulation,  performance  measurement,  syllabus  control  and  record 
keeping  combine  to  produce  a  nearly  " instructor! ess"  training  system. 

The  evaluation  of  ACE  is  not  completed  at  the  time  of  this  writing. 
However,  early  indications  are  that  the  capability  of  the  connected  speech 
recognition  system  to  correctly  recognize  digit  strings,  e.g.,  "two  seven 
zero  twenty  three,"  must  be  improved  to  achieve  full  system  effectiveness 
(McCauley,  Root,  and  Muckier,  in  Preparation).  It  is  possible  that 
improvement  could  be  obtained  by  further  development  of  the  "understanding" 
software  in  support  of  the  speech  recognition  device. 

LANDING  SIGNAL  OFFICER  TRAINING  SYSTEM  (LSOTS) 

The  LSOTS  represents  the  third  NAVTRAEQUIPCEN  program  to  apply  AST  to 
training.  This  system  is  in  the  early  development  stages.  Only  the 
functional  design  and  description  of  the  instructor  model  and 
pilot/aircraft  model  have  been  developed  (Hooks,  Butler,  Gullen,  and 
Petersen,  1978;  Hooks  and  McMurry;  1981;  McCauley  and  Borden,  in  press; 
McCauley,  Cotton  and  Hooks,  in  press).  A  preliminary  test  of  the  concept 
of  real-time  voice  interaction  between  an  LSO  and  *  simulated 
pilot/aircraft  during  carrier  approach  led  to  mixed  results  (Hooks.  Butler, 
Reiss,  and  Petersen,  1980). 

Large  challenges  must  be  faced  in  the  application  of  interactive 
speech  technology  to  LSO  training.  The  LSO's  task  in  carrier  approach 
involves  a  relatively  short  period  of  active  control,  approximately  30 
seconds.  LSO  voice  conmands  must  be  recognized  quickly  (less  than  one 
second)  and  accurately  (approximately  99%)  for  the  system  to  be  effective. 
The  high  stress  of  the  LSO  task  can  be  expected  to  induce  speech 
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variability,  including  differences  in  speech  volume  and  inflection,  that 
will  make  automatic  recognition  difficult.  Perhaps  the  saving  factor  is 
the  relatively  limited  vocabulary,  less  than  70  phrases,  that  could  suffice 
for  an  LSOTS.  Recognition  systems  that  are  "tolerant"  of  stress- induced 
changes  in  speech  will  be  needed  to  support  the  LSOTS  application.  The 
present  authors  do  not  subscribe  to  the  argument  that  the  speech 
recognition  system  can  serve  to  screen  stressed  trainees  from  the  training 
program.  Achieving  a  degree  of  stress  in  the  LSOTS  would  be  evidence  of 
both  good  simulation  and  a  valid  perception  of  the  trainee  that  LSO  errors 
can  be  costly  and  deadly.  The  measure  of  the  LSO  trainee  must  be  that  he 
performs  his  job  well  under  stressful  conditions;  not  that  he  feels  no 
pressure. 

While  the  LSOTS  presents  interesting  challenges,  it  also  epitomizes 
the  potential  benefits  of  AST  in  training  systems.  LSO  training  has  long 
been  accomplished  strictly  by  on-the-job  training.  An  "instructorless" 
LSOTS  would  enable  concentrated  learning  experiences  to  be  provided  without 
placing  a  time  burden  on  the  (always  overworked)  experienced  LSOs  to 
support  the  training  system. 
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SECTION  V 

VOICE  INTERACTIVE  TRAINING  SYSTEM  CONCEPTS 


THE  TRAINING  SYSTEM  DEVELOPMENT  PROCESS 

Automated  speech  technology  can  be  integrated  into  a  training  system 
either  as  a  retrofit  to  an  existing  trainer,  or  as  an  integral  part  of  the 
original  system  design.  This  report  emphasizes  the  latter  application 
because  it  is  more  encompassing  and  more  amenable  to  generic  discussion. 

Training  system  development  should  follow  the  general  guidelines 
of  the  ISD  process  (Branson,  Rayner,  Cox,  Furman,  and  King,  1975).  Media 
selection  is  part  of  the  ISD  process,  but  at  the  present  time,  speech 
technology  is  not  included  in  ISD  documentation  as  a  potential  vehicle  for 
information  exchange  in  training. 

Speech  technology  may  be  appropriate  for  a  training  system  directed 
toward  any  Navy  task  that  involves  speech  output.  The  analysis  of  the  task 
should  include  thorough  documentation  of  the  task  vocabulary,  including  the 
number  of  speakers,  the  size  and  content  of  the  vocabulary,  and  the  use  of 
standarized  vocabulary  and  common  exceptions.  This  vocabulary  analysis 
should  become  a  routine  addition  to  the  task  analysis  of  ISD. 

When  AST  is  considered  for  a  training  system,  the  analysis  of  the 
vocabulary,  the  task,  and  the  training  objectives,  must  be  brought  together 
into  a  top-level  description  of  technological  requirements.  The  state-of- 
the-art  in  AST  is  changing  rapidly.  Therefore,  it  should  be  reassessed  for 
each  training  system  to  determine  whether  it  matches  the  estimated  system 
requirements. 

This  process  involves  a  "top-down"  analysis,  beginning  with  the 
functional  training  requirements.  Too  often,  a  new  technology  is 
identified  and  "force  fit"  into  a  problem.  Automated  speech  technology  has 
very  positive  attributes,  such  as  enabling  real-time  voice  interactive 
simulation  and  automated  scoring  of  speech  tasks.  It  should  be  used,  when 
appropriate,  to  replace  training  support  personnel,  such  as  pseudo-pilots 
or  instructors.  The  decision  to  use  AST  should  be  based  on  cost 
effectiveness,  training  effectiveness  and  manpower  availability. 

The  people  involved  in  developing  a  complex  AST  training  system  should 
reflect  at  least  three  types  of  skills:  1)  a  training  expert 

(instructional  technologist,  educational  system  specialist,  or 
psychologist);  2)  a  systems  engineer  with  experience  in  both  hardware  and 
software;  and  3)  a  subject-matter-expert  (SME)  who  has  recent  experience  in 
both  the  operational  task  and  in  training  the  task.  Another  type  of  person 
who  would  be  a  valuable  asset  for  AST  training  system  development  is  one 
with  background  in  speech  recognition  systems.  Relatively  few  people  fit 
in  this  category  at  present,  and  it  should  not  be  considered  essential  for 
such  expertise  to  be  represented  on  the  design  team.  The  other  team 
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members  will  acquire  speech  system  background  during  the  project.  They 
should  be  encouraged  to  facilitate  this  process  by  assimilating  reviews, 
such  as  Dixon  and  Martin  (1979)  and  Lea  (1980),  and  by  observing  and  using 
actual  speech  systems  whenever  possible. 

As  the  design  team  acquires  skill  in  using  speech  recognition  systems, 
they  should  not  forget  that  the  recognition  performance  they  experience 
will  no  longer  be  representati ve  of  that  obtained  by  the  naive  user,  such 
as  a  new  trainee.  It  is  a  common  tendency  for  system  designers  to  lose 
perspective  of  the  knowledge/skill  differences  between  themselves  and  the 
eventual  users.  Any  system  seems  easy  to  operate  when  one  has  "lived"  with 
it  for  an  extended  time.  The  system  must  be  designed  for  the  eventual 
user,  not  for  the  design  engineer  (i.e.,  education,  background,  skills, 
experience,  and  other  characteristics).  While  these  issues  may  apply 
across  a  broad  spectrum  of  systems,  particular  reference  is  made  here  to 
the  common  finding  that  experienced  speech  recognition  system  users  obtain 
better  recognition  accuracy  than  naive  users  ( cf .  Armstrong  and  Poock, 
1981). 

Management  of  the  system  design  team  can  play  an  important  role  in  the 
characteristics  of  the  final  training  system.  Teamwork  is  essential. 
Overlapping  areas  of  expertise  and  concomitant  responsibility  should  be 
defined  and  discussed.  Balance  between  the  team  members  is  essential  to 
the  efficient  design  of  an  effective  training  system.  Intermittent  contact 
with  Navy  operational  personnel  is  not  sufficient  to  design  a  good  training 
system.  Effective  design  of  automated  instruction  is  promoted  by  daily 
communication  between  the  instructional  specialist,  the  SME,  and  the 
hardware/ software  expert. 

Developing  a  complex  automated  training  system  must  be  an  iterative 
process.  Each  building  block  of  the  system  must  be  developed,  tested, 
evaluated  and  revised  (TEAR)  (see  Figure  3).  The  building  blocks  may  be 
hardware,  firmware,  software,  or  courseware.  As  they  are  integrated  into 
increasingly  larger  conf iguration  items  during  the  system  development,  TEAR 
will  be  necesssary  each  step  of  the  way.  This  concept  is  not  saying 
anything  new  to  system  developers.  But  the  time  and  resources  often  are 
not  allocated  for  the  TEAR  cycle,  particularly  in  the  final  stages  of 
development,  i.e.,  in-plant  testing  and  on-site  (field)  testing. 
Insufficient  time  and  resources  to  accomplish  revisions  at  these  final 
stages  can  compromise  the  effectiveness  of  the  training  system  and  the 
operational  community's  support  for  the  system. 

Because  the  traditional  government  procurement  system  is  based  on 
minimizing  the  competitive  bid,  one  of  the  areas  most  likely  to  suffer  is 
the  TEAR  process  at  the  end  of  the  system  development.  Separating  the 
final  TEAR  stages  from  the  primary  system  development  is  recommended.  This 
procedure  would  assure  the  availability  of  sufficient  time  and  resources  to 
make  the  revisions  before  the  system  is  submitted  to  an  operational  test 
and  evaluation. 
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Figure  3.  The  Training  System  Development  Process  Includes  a 

Continuous  Cycle  of  Develop/Test/Evaluate  and  Revise  (TEAR) 
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The  end  of  the  development  cycle  is  arbitrary.  Delivery  of  an 
operational  training  system  may  be  followed  by  a  final  operational  test  and 
evaluation.  But  the  process  must  be  continued.  Fleet  equipments, 
procedures  and  personnel  change  overtime.  Automated  training  systems  must 
reflect  these  changes.  Scheduled  periodic  review  of  the  match  between 
fleet  training  requirements  and  training  system  capabilities  is  necessary. 
Test  and  revision  should  continue  throughout  the  life  cycle  of  the  training 
system  to  ensure  that  it  meets  the  changing  needs  of  the  fleet. 

DEGREE  OF  INSTRUCTIONAL  AUTOMATION 

In  the  past,  instructors  were  essential  in  the  training  of  tasks  with 
speech  output.  Even  though  some  instructional  materials  could  be  presented 
without  an  instructor  (i.e.,  textbook  or  CAI),  the  trainee's  performance 
(speech)  had  to  be  evaluated  by  the  instructor  listening  to  it. 

Similarly,  pseudo-pilots  were  essential  for  interactive  simulations  of 
tasks  wherein  the  controller  of  some  system  or  vehicle  is  given  verbal 
instructions  by  the  trainee.  In  many  Navy  training  settings,  both  an 
instructor  and  a  support  person  (e.g.,  pseudo  pilot)  are  dedicated  to  the 
training  of  a  single  trainee  (see  Figure  4). 


Figure  4.  Typical  Training  Situation  With  Two  People 
Supporting  The  Instruction  Of  One  Trainee 


Automated  speech  recognition  has  provided  new  opportunities  to  reduce 
the  level  of  instructional  personnel  manloading  by  the  partial  (or  total) 
automation  of  the  pseudo  pilot,  the  instructor,  or  both.  Partial 
automation  can  mean  a  reduction  of  the  number  of  student.,  per  instructor. 
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One  of  the  challenges  facing  the  training  system  design  team  is  to 
determine  the  optimum  degree  of  automation  for  a  particular  training 
system.  Instructor  models  are  becoming  increasingly  capable  of  adequately 
performing  instructor  functions  such  as: 


o  Instruction  (teaching) 
o  Instructional  Sequence  Control 
o  Scenario  Generation 
o  Performance  Measurement  and  Evaluation 
o  Performance  Feedback 
o  Record  Keeping 


Several  NATRAEQUIPCEN  reports  have  contributed  to  the  conceptual 
development  of  automated  instructor  models  (Chatfield  and  Gidcumb,  1977; 
Chatfield,  Marshall,  and  Gidcumb,  1979;  Chatfield,  Klein,  and  Coons,  in 
press;  McCauley  and  Cotton,  in  press).  Advancements  in  the  field  of 
artificial  intelligence  (AI)  are  beginning  to  contribute  to  the 
"intelligence"  of  instructor  models.  The  authors  project  that  by  the 
mid-to-late  1980s  the  application  of  intelligent  instructor  models  will  be 
well  within  the  technology.  The  level  of  "intelligence"  in  automated 
systems  will  evolve  during  the  next  decade.  This  evolution  should  result 
in  the  capability  to  apply  very  effective  automated  instructor  models. 

The  development  of  instructor  models,  however,  is  expensive  because  of 
the  software  required.  The  challenge  to  the  training  systems  development 
team  is  to  weigh  the  software  development  costs  against  the  reduction  in 
instructor  manpower  costs  over  the  projected  life  cycle  of  the  system. 
This  process  assumes  that  equivalent  training  effectiveness  will  be 
achieved  with  human  instructors  and  instructor  models.  However,  an 
exception  to  this  line  of  reasoning  must  be  noted.  In  times  of  absolute 
manpower  shortages,  instructor  models  can  reduce  instructor  manning  levels, 
freeing  instructors  for  fleet  duty.  In  such  cases,  the  cost  of  software 
model  development  may  be  secondary  to  meeting  personnel  manning 
requirements. 

The  strengths  and  weaknesses  of  instructor  models  relative  to  human 
instructors  should  be  considered  when  making  decisions  about  the  degree  of 
automation  desired  in  a  training  system.  Automated  performance  measurement 
and  evaluation  provide  the  potential  benefit  of  complete  objectivity. 
Trainees  need  not  be  concerned  about  possible  bias  of  an  individual 
instructor.  Similarly,  variance  between  instructors  Is  eliminated  with  an 
Instructor  model.  Performance  criteria  are  standardized  and  do  not 
fluctuate  due  to  time  and  environmental  influences. 

Automated  performance  monitoring  provides  increased  capability  for 
record  keeping.  A  large  number  of  trainee  performance  measures  can  be 
recorded  and  tracked  automatically  during  the  training  course.  The  extent 
of  this  performance  data  monitoring  task  would  be  beyond  the  scope  of  the 
human  instructor. 
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On  the  other  hand,  experienced  human  instructors  are  capable  of 
integrating  subtle  cues  to  a  trainee's  state  of  learning.  Human 
instructors  provide  a  source  of  personal  communication  that  seems  to  be 
important  for  some  students  in  a  learning  situation.  While  a  natural 
language  interface  may  be  possible  in  future  training  systems,  it  will  not 
approach  the  richness  and  flexibility  of  human  dialog.  Joplin  (1980)  has 
discussed  the  need  for  supplementing  automated  instruction  with  personal 
interaction  among  students  and  instructors.  Finding  the  optimum  level  of 
automation  for  a  particular  training  application  is  an  important  step  in 
the  ISD  process. 

The  technological  limitations  of  AST  instructor  models  can  be 
identified  in  three  areas  -  speech  recognition,  trainee  models,  and 
performance  measurement/eval uation/diagnosi s. 

Current  limitations  in  automated  speech  recognition  include 
constraints  on  vocabulary  size  and  diminished  recognition  accuracy  under 
"field"  conditions.  These  constraints  represent  a  limitation  of  the 
effective  interchange  between  a  trainee  and  an  instructor  model. 

Every  experienced  human  instructor  uses  his  backlog  of  experience  to 
form  an  internal  model  (or  "schema")  about  how  trainees  should  progress 
during  a  training  course.  The  instructor  can  identify  a  "fast"  or  "slow" 
trainee  by  comparison  to  this  schema.  Instructor  models  can  simulate  a 
consensus  schema  of  normal  (or  abnormal)  progress,  called  a  trainee  model 
(or  student  model),  but  the  training  efficiency  of  the  model  wili  not 
necessarily  exceed  that  of  a  good  instructor.  Wooldridge,  Vreuls,  and 
Norman  (197/)  tested  two  instructors  and  several  automated  adaptive  logics 
and  found  that,  while  they  varied  in  efficiency,  the  best  logics  were 
approximately  equivalent  to  the  better  instructor. 

Instructor  models  can  provide  automated  performance  measurement  of 
many  variables  on  each  behavior  of  the  trainee  (speech  or  other).  However, 
the  development  of  valid  performaince  measures  in  a  difficult  task  is 
fraught  with  pitfalls  such  as  what  variables  to  measure,  when  to  measure 
them,  how  to  score  them  relative  to  some  constant  or  varying  criteria,  and 
how  to  combine  the  scores  into  a  meaningful  set  of  performance  indices 
(Vreuls  and  Wooldridge,  lr'!7).  The  difficulties  with  a  prototype 

performance  measurement  system  in  PARTS  have  been  documented  (McCauley, 
Helms  and  Semple,  1980).  Ineffective  automated  performance  measurement 
places  constraints  on  the  overall  effectiveness  of  an  automated  training 
system.  Although  it  may  not  be,  strictly  speaking,  a  technological 
limitation,  the  development  of  good  automated  performance  measures  is  a 
difficult,  time-consuming  process.  It  requires  continual  interchange  with 
the  operational  training  community  and  a  series  of  TEAR  cycles. 

ANALYSIS  OF  SPEECH  TASKS  FOR  TRAINING 

The  three  AST  training  systems  sponsored  by  NAVTRAEQUIPCEN  have  been 
developed  for  tasks  which  share  comnon  speech  functions.  This  category  of 
tasks  may  be  the  primary  candidate  for  AST  appl  ications;  but  any  task 
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involving  speech  common i cat  ions  could  be  appropriate.  The  three  tasks, 
GCA,  A1C,  and  LSO,  essentially  involve  air  traffic  control.  The  trainee 
must  learn  to  extract  visual  information  and  provide  speech  output  to  a 
pilot.  The  speech  output  represents  the  primary  action  to  be  learned, 
although  both  the  A1C  and  LSO  task  involve  manual  actions  too.  For  all 
three  tasks,  simulation  in  support  of  training  normally  requires  both  an 
instructor  and  a  person  to  simulate  the  actions  of  the  pi  1  ot/ai rcraft . 
These  kinds  of  tasks  are  prime  candidates  for  an  AST  training  system 
because  of  their  emphasis  on  speech  as  the  behavior  to  be  learned,  and 
their  requirement  for  two  people  to  support  the  training  of  each  trainee. 

Navy  training  tasks  vary  in  the  degree  of  speech  behavior  to  be 
learned  i.e.,  what  proportion  of  the  response  repertoire  is  verbal?  The 
decision  about  when  to  use  AST  in  a  training  system  may  be  based  partly  on 
this  factor.  An  anlysis  of  the  verbal  communication  requirements  of  a  task 
is  essential  before  deciding  whether  AST  should  be  a  candidate  technology 
for  a  training  system.  The  analysis  should  include  definition  of  the 

vocabulary  characteristics.  Standardized  phases  of  brief  duration  are  more 
amenable  to  present  AST  than  are  unstructured  and  lengthy  verbiage 

(Doddington  and  Schalk,  1980;  Lea,  1980). 

Team  communication  is  another  candidate  for  AST  training  systems. 

Members  of  the  team  with  whom  the  trainee  must  communicate  can  be  simulated 
by  speech  recognition,  modeling,  and  speech  generation.  This  application 
would  be  helpful  when  actual  team  members  were  not  readily  available  to 
support  the  training  scenario,  or  when  training  effectiveness  could  be 
enhanced  by  greater  control  over  the  actions  of  the  team  members. 

The  techniques  involved  in  the  AST  training  systems  described  above 
are  complex,  but  simpler  training  applications  of  AST  are  possible.  A 

vocabulary  of  four  or  five  words/phrases  could  be  used  to  replace  most 
keyboard  functions  in  a  CAT  system.  A  trainee  getting  "hands  on" 
experience  with  equipment  could  be  given  instruction  by  speech  generation. 
The  trainee  could  be  given  voice  control  over  the  instructiona1  selections 
using  a  few  words,  such  as  "Next"  and  "Review."  These  types  of  AST 
applications  to  training  would  be  relatively  simple  and  inexpensive. 

AST  has  the  potential  to  contribute  to  a  variety  of  Navy  training 
situations,  including  air,  surface  and  subsurface.  The  following  examples 
are  based  on  the  premise  that  any  training  where  verbal  communication  is 
involved  represents  a  candidate  application  of  AST: 

SONAR  team  training 
CIC  training 
Docking  training 
LAMPS  back-up  training 
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AST  also  is  applicable  to  non  speech  (or  low  speech)  tasks  for 
providing  interactive  audio  instruction.  Maintenance  tasks  would  be 
candidates  for  providing  "hands-on"  training  with  interactive  speech 
systems. 

COMMON  PITFALLS 

Inadequate  Trainee-Oriented  Design 

The  functional  characteristics  of  the  interaction  between  the  trainee 
and  the  system  must  be  expressed  thoroughly  before  software  development 
begins.  Human  factors  issues  should  be  emphasized  in  t.he  functional 
description.  For  example,  the  temporal  character!' sties  of  the  interactive 
simulation  must  be  stated,  i.e.,  after  the  trainee  terminates  an  utterance, 
the  system  will  respond  within  one  second. 

In  the  ACE  system,  for  example,  three  to  five  seconds  may  elapse 
before  the  simulated  pilot  responds  to  the  trainee's  verbal  transmission. 
This  has  been  cited  by  ACE  trainees  and  instructors  as  an  unacceptable 
response  time,  (McCauley,  Root  and  Muckier,  in  press).  For  example,  the 
trainee's  task  may  require  him  to  give  a  series  of  transmissions  within  a 
limited  time.  A  delayed  pilot  response  increases  time  pressure  on  the 
trainee.  A  specific  example  from  the  ACE  system  is  given  in  Table  I  to 
illustrate  the  cumulative  effects  of  inadequate  system  response  time.  As 
shown  in  the  table,  slow  response  becomes  an  even  greater  problem  when  it 
is  coupled  with  a  speech  recognition  error.  The  trainee  attempts  to 
transmit  a  series  of  advisories,  some  of  which  require  a  verbal  response 
from  the  pilot.  System  delays  in  providing  vhe  pilot  response  make  tho 
task  more  difficult.  The  delays  are  doubled  when  a  recognition  error 
forces  the  trainee  to  repeat  the  transmission.  A  rare  misrecognition 
requiring  a  repeat  would  be  acceptable  because  pilots  also  may  mi srecogni ?e 
transmissions  occas ional ly.  But  a  high  frequency  of  mi srecogni zed  digi' 
strings  compounds  the  problem  of  a  delayed  verbal  response  from  the  pilot. 

Initial  system  design  goals  should  include  optimized  temporal 
interaction.  Some  delays  will  be  unavoidable  because  of  system 
constraints,  but  high  priority  should  be  given  to  the  timing 
characteristics  of  trainee/system  interaction. 

Inadequate  SME  Input 

Many  Navy  jobs  are  becoming  increasingly  complex  as  technology 
evolves.  Special i zation  is  becoming  the  norm.  The  vocabulary  (or  jargon) 
associated  with  specialized  tasks  is  important  and  often  reflects  subtle 
aspects  of  the  expert's  task  performance.  Training  experts,  system 
analysts,  and  software  programmers  cannot  be  expected  to  master  the 
subtleties  of  a  task  for  which  they  are  developing  a  training  system.  This 
is  a  case  where  a  little  knowledge  can  be  dangerous.  Superficial  under¬ 
standing  of  the  task  will  be  obtained  by  those  working  on  the  project.  A 
SME  with  recent  operational  and  training  experience  should  be  closely 
involved  with  the  development  of  all  aspects  of  the  instructional  material, 
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TABLE  1.  INADEQUATE  TEMPORAL  RELATIONSHIP  OF 
TRAINEE/SYSTEM  VOICE  INTERACTION 


TIME  (SEC.) 

CORRECT  TIMING 

PROTOTYPE  SYSTEM  TIMING 

00 

TRAINEE: 

"Silverhawk, 
Port  130 
for  station" 

TRAINEE:  "Silverhawk, 

Port  130 
for  station" 

01 

02 

SYSTEM: 

"Roger  130" 

03 

04 

TRAINEE: 

"Silverhawk, 
What  State?" 

05 

SYSTEM:  "Roger  180” 

06 

TRAINEE:  "Silverhawk, 

Port  130 
for  station" 

07 

08 

09 

10 

SYSTEM:  "Roger  130" 

11 

TRAINEE:  "Silverhawk, 

What  State?" 
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course  content,  training  objectives,  practice  exercises,  and  performance 
measurement.  Periodic  cross  checks  with  a  larger  sample  of  operational 
SMEs  is  recommended. 

Even  subtle  connotations  of  words  used  in  instruction  can  contribute 
to  the  "not  invented  here"  syndrome.  For  example,  the  term  "advisory"  was 
used  incorrectly  in  the  ground-control led-approach  instruction  presented 
in  PARTS  (McCauley  and  Semple,  1980).  Close-scrutiny  by  a  dedicated  SME 
can  minimize  errors  in  terminology  and  concept.  Instructor  models  must  be 
"intelligent"  about  all  aspects  of  the  job. 

Inadequate  Developmental  Testing 

The  complexity  of  a  training  system  with  real-time  speech  interaction 
and  automated  Instructor  models  should  not  be  underestimated.  Periodic 
stages  of  test,  evaluation  and  revision  (TEAR)  should  be  scheduled  during 
system  development.  The  later  stages  of  the  development  are  particularly 
important  for  developmental  testing  because  of  the  critical  integration  of 
all  the  models  and  subsystems.  It  is  highly  recommended  that  the  TEAR 
process  include  observation  of  a  small  set  of  people  progressing  through 
the  entire  curriculum.  In-plant  personnel  and  SMEs  are  candidates  for 
these  initial  observations.  After  appropriate  revisions  have  been  made,  an 
actual  trainee  should  be  observed  interacting  with  the  training  system. 

The  critical  final  stages  of  the  TEAR  process  often  are  perfunctory 
because  of  the  desire  to  deliver  the  product  within  a  time  schedule. 
Adequate  time  and  resources  should  be  allocated  for  these  important  stages 
of  system  development  including  the  necessary  revisions.  Complex  training 
systems  must  undergo  this  painful  test  and  revision  stage  to  achieve  the 
desired  training  effecti venesss  and  user  acceptance. 
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SECTION  VI 

HOW  TO  USE  THE  DESIGN  GUIDES 


INTENDED  AUDIENCE 

As  stated  in  the  Introduction,  these  design  guides  are  directed  toward 
training  analysts,  subject-matter-experts ,  and  systems  analysts  to  channel 
and  facilitate  their  joint  efforts  in  developing  detailed  functional 
specifications  for  their  particular  training  system  application.  The 
design  guides  are  expected  to  be  used  by  both  government  and  contractor 
personnel  involved  in  the  procurement  and  development  of  AST  training 
systems. 

MAJOR  DIVISIONS 

The  design  guides  are  divided  into  four  categories:  Automated  Speech 
Systems  (Appendix  A),  Instructor  Model  (Appendix  B),  Simulation  and  Event 
Control  (Appendix  C),  and  System  Integration  (Appendix  D).  The  first 
category  is  the  most  extensive,  since  AST  is  the  focus  of  this  project. 
The  remaining  categories  were  developed  under  the  assumption  that  they  are 
part  of  an  AST  training  system.  Much  of  the  information  in  them,  however, 
is  appropriate  for  non- speech  systems  as  well.  The  recommended  design 
procedure  is  given  in  Figure  5. 

FORMAT 

The  four  design  guides  are  presented  in  the  same  format.  Narrative, 
comments  and  supporting  information  are  presented  first.  Then  the  specific 
information,  data,  and  recommendations  which  make  up  each  design  guide 
follow. 

In  developing  the  design  guidelines  the  authors  had  hoped  to  present 
more  specific  information  than  was  ultimately  possible.  This  difficulty 
occurred  for  two  reasons:  1)  a  scarcity  of  design  data  on  speech  systems 
in  field  appl ications,  and  2)  the  orientation  toward  generic  design, 
applicable  to  a  broad  range  of  training  applications.  The  authors  believe, 
however,  that  when  the  guidelines  are  followed  carefully,  an  effective  AST 
training  system  will  result. 

PREREQUISITES 

These  design  guides  are  intended  to  be  used  when  AST  is  being 
considered  for  inclusion  in  an  automated  training  system.  A  full  ISO 
process  should  be  followed  during  the  system  development,  and  the  early 
task  analysis  stages  should  be  completed  before  using  these  design  guides. 
The  design  guides  should  be  helpful  from  that  stage  through  the  development 
of  the  functional  specifications  of  the  system. 
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AUTOMATED  SPEECH  SYSTEM  DESIGN  GUIDE 
PART  1 

SPEECH  RECOGNITION  SYSTEMS 


INTRODUCTION 

The  speech  system  design  guide  is  divided  into  two  parts.  The  first 
covers  the  design  of  speech  recognition  systems  and  the  second  speech 
generation  (synthesis)  systems.  The  reader  is  reminded  that  this  design 
guide  only  applies  to  the  design  of  those  training  systems  involving  verbal 
interaction  between  the  trainee  and  the  training  device.  The  design  guide 
assumes  that  the  user  has  a  working  knowledge  of  automated  speech 
technology  (AST)  and  detailed  knowledge  of  the  training  task  to  which  it 
will  be  applied. 

The  design  guidelines  for  each  section  are  preceded  by  related 
discussion  and  amplifying  comments. 

This  design  guide  is  intended  to  encompass  the  following  design 
procedure  for  speech  recognition  systems,  as  depicted  in  Figure  A-l: 

1.  Extend  the  ISD  process  to  include  speech  task  analysis  for  a 
planned  training  system  development  endeavor. 

2.  Identify  the  speech  recognition  system  functional  design 
requirements  from  the  task  analysis. 

3.  Update  knowledge  about  the  automated  speech  technology 

state-of-the-art. 

4.  Make  a  technical  projection  of  whether  the  design  requirements  for 
the  speech  recognition  system  can  be  satisfied  by  the  technology 
within  the  time  frame  in  which  the  training  system  will  begin  to 
operate. 

5.  Decide  whether  to  continue. 

6.  Develop  the  speech  system  operating  and  human  factors  design  so 
that  the  operation  of  the  system  can  be  thoroughly  described  for 
all  personnel  involved  with  the  design  and  development  process. 

7.  Develop  the  speech  system  specification  so  that  it  becomes  an 
effective  document  for  the  designers,  developers,  and  users  of  the 
training  system. 
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SPEECH  TASKS 
VOCABUl APT 


SPEECH  RECOGNITION 
STSTEM  FEATURES  l 1ST 


PRELIMINARY  FUNCTIONAL 
DESCRIPTION 


Figure  A-l.  Work  Flow  for  Speech  Recognition 
System  Design 
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At  the  time  of  writing  the  ISD  formal  process,  as  depicted  in  Figure 
A-2,  does  not  include  speech  task  analysis.  However,  suggestions  have  been 
made  with  the  military  community  that  the  ISD  formal  process  be  expanded  to 
include  speech  analysis.  For  the  purpose  of  this  design  guide.  Speech  Task 
Analysis  is  intended  to  include  both  speech  recognition  and  speech 
generation. 

It  is  likely  that  speech  recognition  technology  will  move  rapidly  in 
the  next  five  years  (through  1986).  System  designers  will  need  to 
continually  stay  abreast  of  developments  which  will  assist  them  in  making 
prudent  design  decisions.  However,  if  a  prototype  system  is  envisaged,  the 
speech  recognition  technology  may  be  adequate  for  the  performance  of 
limited  training,  but  more  advanced  technology  may  be  required  for  the 
production  training  system.  Thus,  production  timing  and  technical 
projection  are  critical  system  design  factors  to  be  considered  early  in  the 
design  process. 

In  regard  to  operating  and  human  factors  design,  failure  to  thoroughly 
describe  the  operation  of  the  training  system  early  in  the  design  process 
has  a  very  detrimental  effect  on  system  effectiveness  and  on  the  successful 
acceptance  of  the  product  by  the  user  community. 

TASK  AND  VOCABULARY  ANALYSIS 

Speech  tasks  are  a  subset  of  training  tasks  subject  to  the  formal  ISD 
process.  They  can  be  reduced  to  a  basic  unit  of  verbal  communication 
called  an  utterance,  which  may  consist  of  a  word,  digit,  or  phrase.  There 
are  two  basic  categories  of  utterances:  1)  utterances  to  be  said  by  the 
trainee  in  the  conduct  of  training;  and  2)  utterances  to  be  said  to  the 
trainee  by  the  instructor  (or  others)  during  training. 

Utterance  Types 

The  utterances  may  differ  in  specific  segments  of  training  and  should 
be  identified  accordingly.  The  utterance  can  be  a  single  word,  a  single 
digit,  multiple  words  (phrases),  and  multiple  digits  in  combinations. 
Current  speech  recognition  technology  is  capable  of  recognizing  utterances 
consisting  of  single  words  or  groups  of  words  organized  into  short  phrases. 

Commonly  used  phrases  may  be  similarly  constructed  using  different 
words  and  digits.  Consider: 

"turn  left  180°" 

"turn  left  277°" 

the  phrase  structure  remains  the  same  only  the  digits  change.  Now 
consider: 
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"turn  left  1110" 

"turn  right  277°" 

the  phrase  structure  remains  the  same,  only  one  word  changes.  Commercial 
speech  recognizers  capable  of  handling  from  12  to  1000  utterances  are 
available  to  training  system  designers  at  the  time  of  writing.  The 
capacity  for  the  larger  vocabularies  (over  100  utterances)  often  is 
obtained  using  subsetting  procedures,  wherein  only  some  portion  of  the 
entire  vocabulary  is  active  at  any  one  time. 

Develop  a  Speech  Training  Task  Vocabulary 

Table  A-l  shows  a  task  vocabulary  which  was  developed  for  the  Air 
Controller  Exerciser  (see  the  earlier  discussion  in  Section  IV).  It 
contains  utterances  with  embedded  digits  and  embedded  words  and  was 
developed  from  a  series  of  speech  communications  tasks  used  in  the 
wel 1 -structured  Air  Intercept  Control  Environment.  Table  A-2  shows  a  task 
vocabulary  applicable  to  the  Landing  Signal  Officer  Training  System  (LS0TS) 
(LS0  NAT0PS  Manual,  1975).  In  the  carrier  landing  environment  LSOs  often 
use  alternative  phrasings  that  have  different  word  structure  but  the  same 
meaning.  The  LS0  operating  environment  also  allows  emphasis  to  be  imposed 
on  words  to  indicate  to  the  approaching  pilot  the  degree  of  reaction 
indicated,  such  as  "Power"  or  "Power!!!"  (the  latter  said  with  much 
emphasis).  These  kinds  of  vocabulary  idiosyncracies  present  the  training 
system  analyst  with  a  multi-faceted  problem.  The  successful  operation  of 
the  training  system  will  depend  very  heavily  on  satisfactory  solutions. 
Pointers  for  the  vocabulary  development  are: 

o  Categorize  the  speech  tasks  into  operational  segments  (subsetting). 
Identify  singularities  and  commonalities. 

o  Where  a  phrase  differs  only  by  the  use  of  one  word,  create  two 
separate  phrases. 

o  Seek  alternative  means  for  using  the  same  utterance  with  differing 
degrees  of  emphasis. 

o  Seek  alternative  words  to  those  which  have  acoustically  similar 
structur  (like  "niner"  and  "fiver"  or  "port"  and  "fort"). 

o  Use  the  services  of  a  linguist  who  is  familar  with  the  speech 
recognition  technology  to  review  the  proposed  vocabulary  for 
acoustical  confusibi 1 ity.  If  conflicts  occur,  find  alternative 
utterances. 

o  Finalize  the  vocabulary  with  the  user  community.  This  step  also  may 
aid  in  standardizing  the  vocabulary  for  both  the  training  and 
operational  environments. 
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TABLE  A-l.  SPEECH  TASK  VOCABULARY  FOR 
AIR  CONTROLLER  EXERCISER 


The  following  notations  are  used: 


xxx 

yy 

fff 

zz 

m 

n 

c/s 

** 


-  l 


s  the  heading  or  bearing  spoken  as  single  digits 
is  the  range  spoken  as  a  whole  number 
is  the  fuel  spoken  as  a  single  digit 
is  the  altitude  spoken  as  a  whole  number 
is  the  speed  (mach)  as  a  single  digit 
is  a  single  digit  number 
Aircraft  call  sign 

Phrase  may  be  eliminated  to  improve  speech  recognition 


1  =  AIC1 

2  =  ROGER 

3  =  RUTH 

4  =  SAY  AGAIN 

5  =  CORRECTION  ** 

6  =  DISREGARD  THIS  TRANSMISSION** 


PHRASE  NO.  PHRASE 

7  =  BOGEY  (pause)  xxx  (pause)  YY 

8  =  STATION  (pause)  xx  (pause)  YY 

9  =  BOGEY  TRACKING  (pause)  xxx 

10  =  BOGEY  TRACKING  (pause)  xxx  (pause)  SPEED  (pause)  POINT 

11  =  c/s  PORT  (pause)  xxx 

12  =  c/s  STARBOARD  (pause)  xxx 

13  =  c/s  VECTOR  (pause)  xxx 

14  =  c/s  PORT  HARD  (pause)  xxx  ** 

15  =  c/s  STARBOARD  HARD  (pause)  xxx  ** 

16  =  c/s  VECTOR  HARD  (pause)  xxx  ** 

17  =  c/s  PORT  (pause)  xxx  (pause)  FOR  BOGEY  ** 

18  =  c/s  STARBOARD  (pause)  xxx  (pause)  FOR  BOGEY  ** 

19  =  c/s  VECTOR  (pause)  xxx  (pause)  FOR  BOGEY  ** 

20  =  c/s  MARK  YOUR  TACAN 

21  =  c/s  WHAT  STATE 

22  =  ROGER  STATE  (pause)  fff 

(to  CAP) 

(fff  is  fuel  in  hundreds  of  pounds) 

23  =  I  HAVE  CONTROL  OF  c/s 

(to  SWC) 

24  =  c/s  STATE  (pause)  xxx 

(to  SWC) 


(continued) 
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TABLE  A-l.  (Continued) 


25  =  c/S  ON  STATION 

(to  CAP) 

26  =  BOGEY  SINGLE  (pause)  ALTITUDE  (pause)  zz  THOUSAND 

27  =  c/S  BREAKING  AWAY 

28  =  SPLASH  (pause)  n  BOGEYS 

29  =  HEADS  UP  (pause)  n  BOGEYS 

30  =  BOGEY  JINKING  LEFT 

31  =  BOGEY  JINKING  RIGHT 

32  =  BOGEY  SPLITTING 

33  =  ROGER  YOUR  BOGEY  TRACKING  (pause)  xxx 

34  =  NEGATIVE  BOGEY  (pause)  xxx  (pause)  YY 

35  =  BOGEYS  MULTIPLE  (pause)  ALTITUDE  (pause)  zz  THOUSAND 

36  =  STRANGER  (pause)  xxx  (pause)  YY 

37  =  STRANGER  TRACKING  (pause)  xxx 

38  =  STRANGER  TRACKING  (pause)  xxx  (pause)  ANGELS  (pause)  zz 

39  =  STRANGER  OPENING 

40  =  c/s  EASE  TURN 

41  =  c/s  TIGHTEN  TURN 

42  =  c/s  (pause)  xxx  (pause)  YY 

43  =  c/s  ANGELS  (pause)  zz 

44  =  c/s  PORT  (pause)  xxx  (pause)  FOR  RENDEZVOUS  ** 

45  =  c/s  STARBOARD  (pause)  xxx  (pause)  FOR  RENDEZVOUS  ** 

46  =  c/s  VECTOR  (pause)  xxx  (pause)  FOR  RENDEZVOUS  ** 

47  =  c/s  RADIO  CHECK 

48  =  BOGEY  IN  THE  DARK 

49  =  CAP  IN  THE  DARK 

50  =  c/s  MY  OCTOPUS  IS  BENT 

51  =  c/s  EMERGENCY 

52  =  c/s  DETACH  PORT  (pause)  xxx  (pause)  FOR  SEPARATION  ** 

53  =  c/s  DETACH  STARBOARD  (pause)  xxx  (pause)  FOR  SEPARATION 

54  =  c/s  CONTINUED  (pause)  xxx 

55  =  c/s  BREAKAWAY  (pause)  xxx 

56  =  c/S  ANCHOR  PORT 

57  =  c/s  ANCHOR  STARBOARD 

58  =  c/s  STEADY 

59  =  c/s  LOST  COMMUNICATIONS  INTENTIONS 

60  =  ROGER  LOST  COMMUNICATIONS  INTENTIONS 

61  =  c/s  PORT  (pause)  xxx  (pause)  AS  BOGEY  ** 

62  =  c/s  STARBOARD  (pause)  xxx  (pause)  AS  BOGEY  ** 

63  =  c/s  VECTOR  (pause)  xxx  (pause)  AS  BOGEY  ** 
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TABLE  A-2.  SPEECH  TASK  VOCABULARY  FOR 
LANDING  SIGNAL  OFFICER 


INFORMATIVE  CALLS 

Used  to  inform  pilots  of  existing  situations. 


TRANSMISSION 

MEANING 

“ 

RESPONSE  (Aircraft 
in  Manual  Mode) 

RESPONSE  (Aircraft 
in  APC  Mode) 

“You’re  (a  little) 
high.” 

Aircraft  is  (slightly) 
above  optimum  glide- 
slope. 

Adjust  sink  rate  with 
power/nose  attitude  to 
establish  center  ball. 

Adjust  sink  rate  with 
nose  attitude  to  establish 
center  ball.  (Avoid 
using  in  close.) 

“You’re  (a  little) 
low.” 

Aircraft  is  (slightly) 
below  optimum  glide- 
slope. 

Adjust  altitude 
immediately. 

Adjust  altitude  immedi¬ 
ately. 

“You’re  going  high 
(low).” 

Unless  corrected,  air¬ 
craft  will  go  above 
(below)  optimum  glide- 
slope. 

Adjust  sink  rate  with 
power/nose  attitude  to 
maintain  center  ball. 

Adjust  sink  rate  with 
nose  attitude  to  maintain 
center  ball. 

“You’re  lined  up 
left/right.” 

Aircraft  has  undershot/ 
overshot  centerline. 

Reestablish  centered 
lineup. 

Reestablish  centered 
lineup. 

“You’re  drifting 
left/right.” 

Aircraft  is  drifting 
left/right  of  center- 
line. 

Correct  lineup  to 
centerline. 

Correct  lineup  to 
centerline. 

“You’re  fast/slow.” 

Self  explanatory. 

Adjust  nose  attitude/ 
power  to  establish 
optimum  AOA. 

Not  used. 

“Roger  Ball” 
(AUTO/MANUAL/ 
Coupled  as 
appropriate) 

LSO  acknowledges 
pilot  meatball 
acquisition. 

“Paddles  Contact” 

LSO  assuming  control 
from  CCA. 

(continued) 
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TABLE  A-2.  (Continued) 

IMPERATIVE  CALLS 

Used  to  direct  pilot  to  execute  a  specific  control  action.  MANDATORY  IMMEDIATE  RESPONSE. 


TRANSMISSION 

MEANING 

RESPONSE  (Aircraft 
in  Manual  Mode) 

RESPONSE  (Aircraft 
in  APC  Mode) 

“A  little  power." 

Aircraft  is  decel¬ 
erating;  unless  cor¬ 
rected  aircraft  will 
become  slow/low. 

Correct  with  power. 

Call  not  used. 

“Power.” 

Aircraft  is  low/slow. 

Add  power. 

Add  power  and  disengage 
APC.  Refer  to  Note. 

“Go  Manual.” 

Disengage  APC. 

Not  Used. 

Add  power  and  disengage 
APC.  Refer  to  Note. 

“Attitude  ”-( “A  little 

Aircraft  nose  is  low/ 

Increase  nose  attitude 

Increase  nose  attitude 

attitude.”) 

flat  attitude. 

(slightly). 

(slightly)  to  reduce  sink 
rate. 

“Right/Left  for 

Aircraft  will  land 

Correct  lineup  to 

Correct  lineup  to  center 

lineup”  (Use  in  close 
or  at  the  ramp.) 

left/right  if  not 
corrected. 

centerline,  then  level 
wings. 

line,  then  level  wings. 

“Bolter.” 

Self  explanatory. 

Add  100  percent  power 
and  execute  bolter  in 
accordance  with  model 
NATOPS  manual. 

Add  100  percent  power 
and  execute  bolter  in 
accordance  with  model 
NATOPS  manual. 

“Waveoff”  or 
“Waveoff,  Foul 
deck”  (Whenever 

Self  explanatory. 

Execute  waveoff  in 
accordance  with  model 
NATOPS  manual. 

Execute  waveoff  in  ao 
cordance  with  model 
NATOPS  manual. 

waveoff  lights  are 
keyed.) 

“Cut.” 

Release  signal,  as 
necessary  to  landing. 

Response  mandatory 
for  all  prop  landings 
and  jet  barricade 
engagements. 

Response  mandatory  for 
barricade  engagements. 

“Speedbrakes.” 

Speedbrakes  are 
extended. 

Retract  speedbrakes. 

Retract  speedbrakes. 

“Extend  Speed- 

Self  explanatory. 

Comply. 

Comply. 

brakes.” 

“Drop  your  hook.” 

II 

•  i 

“Drop  your  gear.” 

•  l 

li 

II 

"Drop  your  flaps." 

II 

ii 

ll 

Uncouple. 

Disengage  ACLS. 

Disengage  ACLS 

Disengage  ACLS. 
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TABLE  A-2.  (Continued) 


PRECAUTIONARY  CALLS 

Used  to  direct  pilot’s  attention  to  potential  difficulties  and  prevent  possible  control  errors. 


TRANSMISSION 

MEANING 

RESPONSE  (Aircraft 
in  Manual  Mode) 

RESPONSE  (Aircraft 
in  APC  Mode) 

“Check  your  lineup.” 

Aircraft  lineup  is  not 
optimum. 

Correct  lineup  drift  or 
position. 

Correct  lineup  drift  or 
position. 

“Don’t  settle”- 
“Don’t  go  low.” 

Aircraft  will  settle 
below  optimum  glide- 
slope  if  not  corrected. 

Check  sink  rate  and 
meatball  to  avoid 
settling  below  glide- 
slope. 

Check  sink  rate  and  meat- 
ball  to  avoid  settling 
below  glideslope. 

“Don’t  climb”- 
“Don’t  go  high.” 

If  not  corrected  air¬ 
craft  will  climb  above 
optimum  glideslope. 

Check  sink  rate  and 
meatball  to  avoid 
climbing  above  glide- 
slope. 

Check  sink  rate  and  meat- 
ball  to  avoid  climbing 
above  glideslope. 

“Keep  your  nose  up”- 
“Hold  your  attitude.” 

Pilot  tends  to  drop 
nose. 

Don’t  drop  nose. 

Don’t  drop  nose. 

“Hold  what  you’ve 
got.” 

Self  explanatory. 

Hold  present  (optimum) 
stick  and  throttle 
positions. 

Hold  present  (optimum) 
stick  position. 

(continued) 
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There  may  be  no  simple  solution  to  the  problem  of  emphatic  words  and 
phrases.  However,  recognizing  that  the  majority  of  Isolated  Word/Phrase 
speech  recognizers  are  speaker  dependent,  one  solution  may  be  to  "sample  " 
the  utterance  at  different  degrees  of  emphasis,  i.e.  to  treat  them  as 
different  utterances.  Another  solution  is  to  provide  heavy  support  to  the 
speech  recognizer  in  the  form  of  "understanding"  software  that  makes  use  of 
all  available  task  and  context  cues. 

TIME  CONSTRAINTS 

Recognition  Response  Time 

The  initial  analysis  of  the  speech  recognition  system  must  include 
identification  of  the  time  constraints  in  which  speech  recognition  must 
occur  to  maintain  realistic  pacing  of  the  task.  Approximately  one  second 
is  the  maximum  time  that  should  be  allowed  for  recognition  to  take  place  in 
an  interactive  system.  After  one  second,  the  training  system  should  begin 
to  provide  some  form  of  visual  or  aural  response  indicating  that 
recognition  has  taken  place.  Without  adequate  response  time,  the  user  may 
engage  in  repeated  efforts  to  get  the  training  system  to  respond.  This 
interferes  with  the  training  scenario  and  detracts  from  user  acceptance. 

Pausing  Between  Utterances 

Most  current  recognition  systems  require  the  user  to  pause  between 
each  utterance  for  approximately  100  to  200  milliseconds.  This  pause 
length  is  natural  for  a  human  speaking  in  non-stressful  circumstances. 
However,  if  the  training  circumstance  becomes  stressful  (and  most  will  do 
so  at  some  point),  the  trainee  will  tend  to  omit  pauses,  severely  degrading 
recognition  accuracy.  Recent  progress  in  ASR  technology  promises  to  reduce 
the  minimum  pause  to  50  milliseconds  or  less. 

Phrase  Length 

Current  recognizers  require  a  vocabulary  item  to  be  input  within  some 
maximum  time  limit.  This  limit  varies  with  the  manufacturer,  but  most  are 
1  1/2  to  5  seconds.  No  pause  can  occur  between  the  words  of  a  phrase 
defined  as  a  vocabulary  item. 

DEFINING  THE  SPEECH  RECOGNITION  SYSTEM  REQUIREMENTS 

The  requirements  of  the  speech  recognition  system  must  be  specified 
for  each  training  system  application.  Table  A-3  lists  the  potential 
requirements  which  should  be  considered.  For  ease  of  use  the  table  has 
been  subdivided  as  follows: 

Speech  Characteristics  -  the  task-oriented  vocabulary. 

Functional  Design  Features  -  how  the  system  accomplishes  the 
job  of  recognition. 
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Table  A-3.  SPEECH  RECOGNITION  SYSTEM  POTENTIAL  REQUIREMENTS 


A  SPEECH  CHARACTERISTICS 

Continuous  Speech  Recognition 

Connected  or  Limited  Continuous  Speech  Recognition 
Isolated  Word  (Phrase)  Recognition 
Vocabulary  Size 
Subsetted  Vocabulary  Size 
Stylization  Requirements 
B  FUNCTIONAL  DESIGN  FEATURES 
Speech  Data  Collection 
Speech  Recognition  Practice 
Recognition  Test 
Confusibil ity  Index 
Repeated  Speech  Data  Collection 
Speech  Recognition  Feedback 
Syntax  Control 
Under standi ng 
Multiple  Concurrent  Users 
Methods  for  Signing  On 
Speech  On/Off  Control 


(continued) 
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Table  A-3.  (Continued) 


C  RECOGNITION  PERFORMANCE  CHARACTERISTICS 
Speaker  Independent/Dependent 
Transmission  Accuracy 
Rejection  Error 
Speaking  Rates 
Pause  Duration 

Reaction  Time  (after  completion  of  transmission) 

Rejection  Thresholds 

User  Variabilites 

Environmental  Perturbations 

D  MISCELLANEOUS  FEATURES 

Projected  Cost 

Speech  Input  Device 
type  of  microphone 

Speech  Input  Level  Control 

automatic  gain  control,  manual  gain  control  with  Vu  meter 
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Recognition  Performance  Characteristics  -  how  the  speech 
recognition  systems  performs. 

Miscellaneous  Features  -  not  covered  within  the  foregoing 
categories. 


Speech  Characteristics 

Continuous  Speech  Recognition  -  is  characterized  by  a  large  vocabulary, 
long  utterances  and  few  pauses. 

Connected  (or  Limited  Continuous)  Speech  Recognition  -  allows  a  series  of 
discrete  words  or  phrases  to  be  spoken  without  pausing  in  a  maximum  time 
span  (normally  2-5  seconds).  Similarly,  digit  strings  can  be  recognized 
without  pausing  between  the  digits. 

Isolated  Word  (Phrase)  Recognition  -  requires  pauses  between  discrete  words 
and/or  phrases  which  must  be  no  longer  than  within  a  maximum  time  span 
(normally  1  1/2  -  5  seconds).  A  digit  string  which  is  used  repeatedly, 
like  "BRAVO  129",  is  considered  to  be  a  discrete  word.  Digit  strings  which 
vary  will  require  pausing  between  the  digits,  like  "TURN  TO...  2. ..7. ..5... 
DEGREES." 

Vocabulary  Size  -  can  be  up  to  several  hundred  words  and  or  phrases,  or 
multiplied  to  larger  vocabularies  by  subsetting. 

Subsetted  Vocabularies  -  vocabularies  can  be  partitioned  into  subsets  as  a 
function  of  the  operating  environment  (sometimes  referred  to  as  task 
syntax  or  task  oriented  grammar).  This  is  usually  done  as  a  means  of 
limiting  the  vocabulary  size  which  must  be  actively  processed.  Subsetting 
can  be  us^d  either  to  improve  recognition  accuracy  through  decreasing  the 
active  vocabulary  or  to  increase  the  maximum  vocabulary  size. 

Stylization  -  refers  to  the  constraints  imposed  on  the  speaker  by  the 
recognition  system  in  regard  to  pausing,  volume,  and  pronunciation.  Proper 
pausing  is  essential  for  isolated  word  recognition.  The  pause  requirements 
imposed  by  the  recognition  process  must  be  consistent  with  the  operational 
language.  Stylization  constraints  are  a  reflection  of  limitations  in 
current  recognition  technology.  As  the  technology  advances,  stylization 
constraints  will  be  reduced. 

Function  Design  Features 

Speech  Data  Collection  -  takes  a  sample  of  the  speech  for  the  defined 
vocabulary  spoken  Ey~  each  user  of  a  speaker  dependent  system.  These 
samples  become  the  template  or  reference  pattern  for  all  subsequent 
recognition. 
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Speech  data  collection  is  conducted  when  a  trainee  first  uses  the 
system,  when  a  new  vocabulary  word  or  phrase  is  added,  and  when  speech 
recognition  performance  becomes  degraded. 

Speaker-dependent  recognizers  currently  require  2-10  samples  for 
each  word/phrase.  In  some  systems,  part  of  the  original  sample  set  should 
be  updated  each  time  the  trainee  signs-on.  On  other  systems,  sample 
updating  is  rarely  required.  Future  speech  recognition  systems  are 
expected  to  optimize  the  vocabulary  speech  pattern  information  by 
periodically  updating  the  sample  with  correctly  recognized  words/phrases 
uttered  during  the  conduct  of  training. 

Samples  should  be  collected  in  the  context  of  the  task(s)  to  be 
learned.  Visual  prompts  are  preferable  to  audio  prompts  for  speech  data 
collection  to  preclude  the  trainee  emulating  the  characteristics  of  the 
speech  generation  system. 

Speech  Recognition  Practice  -  facilities  should  be  provided  for  the  trainee 
to  practice  speech  recognition.  This  will  lead  to  consistent  speech  and 
better  recognition  performance. 

Recognition  Test  -  is  a  means  of  evaluating  speech  recognition  system 
accuracy  indpendent  of  the  training  task.  This  can  be  done  simply  by 
providing  visual  or  audio  information  of  what  has  been  recognized. 

Confusibility  Index  -  is  a  means  of  relating  the  probability  of  recognition 
of  each  word  or  phrase  to  other  words  and  phrases  in  the  vocabulary.  This 
is  a  special  measurement  feature  of  recognition  test.  The  total  matrix  may 
be  displayed,  or,  alternatively,  only  those  words  and  phrases  with  critical 
confusion.  The  confusion  matrix  also  can  be  used  to  prompt  the  retraining 
of  confused  words  and  phrases. 

Repeated  Speech  Data  Collection  -  should  be  used  to  eliminate  confused  or 
rejected  vocabulary  words  and  phrases.  These  conditions  tend  to  occur  when 
there  are  changes  in  speech  characteristics  due  to  fatigue,  stress,  colds, 
flu,  and  changes  in  ambient  noise. 

Speech  Recognition  Feedback  -  is  information  provided  to  the  trainee  (and 
instructor)  about  the  words  and  phrases  which  have  been  recognized. 
Feedback  is  essential  but  it  must  not  interfere  with  the  conduct  of  the 
training  task.  In  some  instances  it  may  not  be  advisable  to  provide 
feedback  until  completion  of  the  immediate  task. 

Syntax  Controls  -  are  rules  which  govern  the  use  of  the  vocabulary  in  an 
operational  context.  In  specific  event  sequences  certain  phrases  have  an 
extremely  high  probability  of  being  correct.  Software  representation  of 
this  knowledge  can  improve  recognition  accuracy. 
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Understanding  -  is  analogous  to  an  artificial  intelligence  process  and 
provides  a  higher  order  knowledge  (above  speech  recognition  which  is  based 
on  acoustic  pattern  matching)  to  determine  what  has  been  said.  "oftware 
representation  of  task  and  linguistic  knowledge  can  improve  .  ..gnition 
accuracy. 

Multiple  Concurrent  Users  -  system  designers  may  have  to  consider  the  use 
of  multiple  recognition  systems  working  with  one  host  computer  to  provide  a 
multiple  station  training  system.  Recognition  for  each  trainee  must  be 
independent  of  all  other  trainees.  Speech  recognition  reaction  time  for  an 
individual  trainee’s  station  should  not  exceed  one  second. 

Methods  of  Signing  On  -  for  speaker  dependent  systems,  the  speech  pattern 
data  of  a  trainee  must  be  bought  into  the  foreground  each  time  he  signs  on 
the  system.  In  addition,  the  trainee's  professional  data,  past  history, 
and  performance  on  the  training  system  also  have  to  be  in  the  foreground. 
This  information  should  be  maintained  on  a  diskette  (or  equivalent)  which 
is  inserted  into  the  training  system  each  time  he  uses  it.  This  diskettte 
becomes  the  trainee's  information  file  during  training. 

Speech  On/Off  Control  -  is  recommended  for  task  oriented  vocabulary 
systems,  as  the  discrete  on/off  action  alerts  the  system  that  speech 
recognition  is  expected  of  it.  Microphone  keys  or  foot  switches  are 
recommended.  Voice  operated  switches  may  be  appropriate  although  caution 
is  recommended  because  of  the  potential  loss  of  valuable  acoustical 
information  at  the  onset  of  the  word  or  phrase. 

RECOGNITION  PERFORMANCE  CHARACTERISTICS 


Speaker  Independent/Dependent  -  the  majority  of  current  speech  recognition 
systems  are  speaker  dependent.  Exceptions  are  those  systems  in  which 
digits,  words  or  simple  phrases  are  said  by  any  speaker,  often  in  response 
to  a  series  of  audio  prompted  questions.  Speaker  dependency  will  continue 
to  be  the  rule  for  training  systems  until  future,  non-task  oriented,  large 
vocabulary,  speaker  independent  recognition  systems  become  available. 

Recognition  Accuracy  -  measures  must  account  for  correct  recognitions,  the 
incorrect  recognitions  and  non-recognitions  (rejections)  of  valid 
vocabulary.  Percent  transmission  accuracy  (TA)  is  hereby  defined  as: 

#  of  valid  -  (#  of  incorrect  +  #  of  non¬ 
transmission  recognitions  recognitions) 

TA  *  X  100 


#  of  valid  transmissions 

A  valid  transmission  refers  to  a  programmed  vocabulary  item  in  the  form  of 
digits,  words,  phrases  and  combinations  thereof. 
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Rejection  Error  -  of  a  speech  recognition  system  must  account  for  the 
non-valid  transmissions  made  by  the  user  and  the  non-valid  recognitions 
made  by  the  system.  Percent  rejection  error  (RE)  is  hereby  defined  as: 

#  of  non-valid  recognitions 
RE  = _ X  100 


#  of  non-valid  transmissions 

A  non-valid  transmission  refers  to  a  non- programmed  vocabulary  item  in  the 
form  of  digits,  words,  phrases,  combinations  thereof,  and  other  utterances 
like  "ah,  umm." 

Speaking  Rates  -  most  current  recognizers  will  accept  an  utterance  with  a 
maximum  duration  (depending  on  the  manufacturer)  and  will  accept  up  to  50 
utterances  per  minute.  Current  advances  in  the  technology  reportedly 
provide  successful  recognition  of  up  to  180  words  per  minute. 

Pause  Duration  -  most  current  systems  require  pauses  between  utterances  to 
be  greater  than  100  milliseconds.  Again,  current  advances  reportedly  have 
reduced  the  pause  duration  to  less  than  50  milliseconds. 

Reaction  Time  -  An  interactive  system  should  respond  within  one  second 
after  completion  of  an  utterance. 

Rejection  Thresholds  -  define  the  relationship  between  correct  recognition, 
incorrect  recogni t ion/word  substitution  error,  or  rejection  of  a 
transmission.  They  are  normally  preset  by  the  manufacturer.  The 
unpublished  industry's  norm  for  recognition/rejection  settings  is 
considered  to  be  95%  correct  recognition  rate,  with  3%  rejection  rate  and 
2%  word  substitution.  In  some  applications,  like  Command  and  Control,  a 
word  substitution  rate  of  .005%  is  the  minimum  acceptance  for  initiating 
critical  commands  whereas  a  10%  rejection  rate  is  operationally  acceptable. 
Thus,  user  adjustment  of  rejection  thresholds  is  desirable. 

Speech  Variability  -  within  a  speaker  can  arise  from  many  sources  including 
stress,  fatigue,  and  health  (cold,  flu,  etc.).  Speech  variabilities  are 
associated  with  decreased  recognition  performance. 

Environmental  Perturbations  -  are  defined  as  environmental  factors  that 
influence  speech  recognition  such  as  ambient  noise,  vibration,  high  G, 
etc.  These  factors  may  be  overcome  to  some  extent  by  repeated  voice  data 
collection  with  the  environmental  disturbance  present. 

Miscellaneous  Features 


Projected  Cost  -  Currently  speech  recognizers  vary  in  cost  from 
approximately  $1,000  to  $100,000  per  unit,  depending  on  capability.  Cost 
comparison  of  systems  is  difficult  because  their  features  and  capabilities 
vary  greatly.  Furthermore,  standardized  performance  specifications  do  not 
exist  at  the  present  time.  Cost  estimates  should  be  based  primarily  on 
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three  factors:  1)  will  the  recognizer  stand  alone  or  will  there  be 

software  to  give  functional  support  to  recognition;  2)  what  type  of  speech 
recognition  is  required  -  isolated  word,  connected  word,  or,  (in  the 
future)  continuous  speech;  and  3)  what  is  the  size  and  structure  of  the 
vocabulary  that  must  be  handled. 

Speech  Input  Device  -  the  system  designer  may  be  confronted  with  a  user 
requirement  that  the  normal  operational  type  of  input  device  be  used,  for 
example,  a  telephone  handset.  Whenever  possible,  a  noise-cancelling  boom 
microphone  should  be  selected  for  its  consistency  of  input. 

Speech  Input  Level  Control  -  automatic  gain  control  of  the  trainee  speech 
audio  input  to  the  recognizer  is  preferable  to  manual  control  with  a  Vu 
meter  because  the  former  method  provides  consistency. 

REASSESSING  THE  SPEECH  TECHNOLOGY  STATE-OF-THE-ART 

Speech  recognition  systems  are  becoming  more  compact  and  more  capable 
with  the  advent  of  medium  to  large  scale  integrated  electronic  component 
technology.  While  the  breakthrough  to  unrestricted  continuous  speech  still 
may  be  years  away,  ongoing  improvements  will  continue  in  isolated  phrase 
and  limited  continuous  speech  recognizers.  These  improvements  will  warrant 
reassessment  of  the  technology  each  time  a  training  system  design  or 
modification  is  considered.  The  objective  here  is  to  update  a  "current" 
technology  baseline.  The  technology  assessment  should  be  carried  out  on  a 
feature-by- feature  basis  using  a  similar  listing  to  that  shown  in  Table 
A-3. 


Each  feature  should  be  related  in  a  systematic  manner  to:  1)  the  most 
recent  experience  gained  with  specific  speech  recognition  systems  operating 
in  the  field,  and  2)  the  latest  claims  made  by  manufacturers  in  regard  to  a 
specific  recognition  product  capability. 


"Hands-on"  exposure  is  a  necessity  to  evaluate  recent  system 
experience  in  the  context  of  the  requirements  of  the  system  under 
development. 

In  regard  to  manufacturers  latest  claims,  one  method  for  comparison 
which  gives  a  view  of  advancing  technology  is  to  review  the  expanding 
capability  of  a  line  of  recognizers  produced  by  the  same  manufacturer  over 
time. 

MAKING  A  SPEECH  RECOGNITION  TECHNOLOGY  PROJECTION 

Having  established  the  current  state-of-technology,  the  training 
system  design  team  is  confronted  with  making  a  projection  of  where  the 
technology  is  likely  to  be  while  the  training  system  is  being  developed. 

At  the  time  of  writing,  this  is  a  difficult  task  because  of  the 
following  factors: 
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1.  Basic  research  and  development  in  the  realm  of  unrestricted 
continuous  speech  is  being  conducted  by  large  corporate  entities 
like  IBM,  Bell  Labs  and  Texas  Instruments,  who  guard  their 
knowledge  until  the  product  is  publicly  introduced. 

2.  In  the  immediate  future,  systems  development  firms  have  the 
potential  to  use  imaginative  computer  programming  to  promote 
comprehensive  task-oriented  speech  "understanding"  systems  while 
using  1979-80  generation  recognizers. 

3.  The  speed  at  which  industry  advances  the  technology  is  directly 
related  to  the  number  of  speech  recognition  system  applications 
being  made  or  updated  in  the  field. 


However,  in  terms  of  voice  interactive  training  system  development, 
the  technology  projection  must  be  completed  before  any  attemps  are  made  to 
determine  whether  a  match  between  speech  recognition  requirements  and  the 
technology  is  feasible.  There  are  six  key  points  on  which  the  projection 
should  be  made.  These  are: 

1.  What  type  of  recognition  mechanization  will  be  available  in  the 
"system  build"  time  frame?  (isolated  word,  limited  connected 
speech,  or  continuous  speech)  . 

2.  What  will  be  the  degree  of  speaker  independence?  (total,  initial 
speech  data  collection  required,  frequent  update  data  collection 
required). 

3.  What  is  the  expected  recognition  (transmission)  accuracy  in  the 
operational  environment? 

4.  Will  there  be  any  timing  or  processing  constraints  imposed  on  the 
training  system  by  the  available  technology? 

5.  Will  the  available  technology  foster  user  acceptance  of  the  speech 
interface? 

6.  Does  the  cost  of  the  speech  recognition  system  exceed  the  training 
system  procurement  budget? 


Milestone  projects  from  NAVTRAEQUIPCEN,  like  PARTS  and  ACE,  have 
evolved  through  prototype  system  developments.  Prudent  training/systems 
analysts  who  begin  with  prototype  system  development,  should  attempt  to 
project  whether  future  technology  will  be  commensurate  with  the  greater 
demands  of  the  production  system. 
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MATCHING  TECHNOLOGY  TO  SYSTEM  REQUIREMENTS 

Based  on  the  definition  of  the  speech  recognition  system  requirements 
and  the  technology  projection,  it  will  be  possible  to  determine  whether 
there  is  a  match.  If  there  is  not  a  complete  match  across  the  spectrum  of 
requirements  listed  in  Table  A-3,  then  the  training  system  requirements 
must  be  reduced  and  the  matching  process  reiterated. 

Practically  speaking,  reiteration  could  compromise  the  system 
requirements  to  such  an  extent  that  the  training  system  would  no  longer 
satisfy  the  user  requirements.  Full  communication  with  the  user  community 
during  such  a  reiteration  process  is  strongly  recommended.  Time  appears  to 
be  on  the  side  of  training/ systems  analysts  because  of  the  expected 
improvements  in  speech  recognition  technology.  If  a  reduced  training 
system  capability  is  agreed  upon,  then  it  should  be  formally  recorded. 
When  the  speech  recognition  technology  projection  will  not  meet  the  system 
requirements,  an  agreement  may  be  possible  to  continue  with  the  training 
system  production  with  a  view  to  adding  speech  recognition  technology 
later.  Such  a  decision  requires  substantial  attention  to  specifying  and 
designing  a  system  that  can  be  retrofitted  and  still  be  cost  effective. 

In  addition  to  the  hazards  arising  from  compromising  the  basic 
training  system  design,  the  training/systems  analysts  must  be  aware  of 
three  other  pitfalls  which  have  their  origin  in  the  technology  matching 
process:  These  are: 

1.  Underprojection  of  the  technology  to  meet  a  specific  speech 
recognition  system  requirement.  This  will  produce  a  system  with 
lesser  capability  than  was  possible.  However,  this  conservative 
approach  can  be  relied  on  to  produce  a  training  system  which  will 
meet  the  requirements. 

2.  Overprojection  of  the  technology  to  meet  the  system  requirements. 
This  can  produce  a  system  where  the  speech  recognition  system  will 
not  fulfill  the  design  requirements,  thus  making  the  training 
system  ineffective. 

3.  Inappropriate  analysis  of  features  requirements  may  cause  the  speech 
recognition  system  to  be  too  sophisticated  or  too  rudimentary  in 
fulfilling  the  training  requirements. 

In  conclusion,  the  output  of  the  matching  process  should  always  be  a 
hard  YES/NO  decision.  If  the  decision  is  YES,  it  should  be  consumated  by  a 
training  system  design  requirement  document  which  includes  the  speech 
recognition  system  and  is  ratified  by  the  user  community. 

SPEECH  SYSTEM  OPERATING  AND  HUMAN  FACTORS  DESIGN 

Once  the  decision  to  proceed  with  system  design  has  been  made,  it  is 
vitally  important  to  describe  exactly  how  the  system  will  operate.  The 
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level  to  which  this  description  is  made  must  fulfill  the  needs  of  the 
working  level  software  programming  staff  as  well  as  the  needs  of  the 
operational  training  manager. 

Training  Scenario  Development 

The  recommended  approach  is  for  the  training/ systems  analyst,  in 
conjunction  with  a  subject  matter  expert,  to  develop  an  operating  scenario 
for  the  overall  training  system.  Each  event,  including  speech,  that  takes 
place  in  the  interface  between  the  training  device  and  the  trainee  (and 
instructor  when  applicable)  should  be  described  in  a  step-by-step  manner 
for  each  part  of  the  scenario.  During  this  process,  the  analysts  will 
discover  a  series  of  human  factors  design  questions  which  have  to  be 
resolved  as  they  arise  in  order  to  continue  with  the  operating  scenario 
development.  The  product  of  the  scenario  developed  is  the  preliminary 
functional  design  which  includes  the  functions  of  the  speech  recognition 
system.  It  must  be  reemphasized  that  each  part  of  the  operating  scenario 
should  be  in  such  detail  that  each  software  programmer  understands  what  is 
required  of  him/her  throughout  the  training  system  operational  picture. 

A  recommended  method  for  structuring  the  development  of  an  operating 
scenario  is  to  partition  the  operation  by  modes  and  submodes.  In  the  case 
of  the  LSOTS  (McCauley  and  Cotton,  in  press))  the  system  operation  was 
partitioned  as  follows: 

o  Initialize 
o  Demonstration 
o  Instructional 
o  Manual  backup 
o  Off-line 

These  modes  were  then  partitioned  into  submodes  as  shown  in  Table 
A-4.  The  system  operation  and  the  manner  in  which  the  trainee  interface., 
with  the  system  was  then  described  for  each  submode.  For  example,  in  LSOTS 
the  speech  recognition  test  submode  can  be  called  on  by  the  trainee  and/or 
the  instructor  during  the  instruction,  practice,  and  debrief  submodes. 
Furthermore,  during  the  development  of  the  LSOTS  operating  scenario,  it  was 
decided  to  introduce  a  manual  backup  mode  in  the  event  that  the  speech 
recognition  system  would  not  operate  with  a  particular  trainee’s  voice 
characteristics. 

Human  Factors  Considerations 

Each  training  system  design  will  require  specific  human  factors 
considerations  which  apoly  to  the  speech  recognition  system  (see  Van 
Hemel  ,  et  al.,  1980).  However,  there  are  three  human  factors 
considerations  which  are  common  to  all  voice  interactive  training  systems 
and  which  the  training/ system  analyst  must  carefully  consider.  These  are: 
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o  Speech  data  collection 
o  Speech  recognition  testing 
o  Recognition  feedback 

Speech  Data  Collection  -  This  is  a  requirement  of  speaker-dependent 
systems.  The  objective  is  to  collect  reference  patterns  about  how  each 
user  says  each  digit/word/phrase  in  the  vocabulary.  It  is  desirable  to 
collect  this  data  in  the  training  context.  However,  this  requires  time  on 
the  system  and  if  the  user  population  is  large  and  time  on  the  system  is 
short,  streamlined  data  collection  procedures  may  be  necessary.  If  the 
total  time  for  speech  data  collection  is  long,  it  may  be  beneficial  to  do 
the  sampling  over  several  sessions  to  avoid  boredom  and  voice  fatigue. 
Thus,  speech  data  collection  procedures  become  an  important  part  of  the 
training  curriculum. 

Current  speaker-dependent  recognizers  require  two  to  ten  samples.  For 
a  vocabulary  of  several  hundred  words,  speech  sampling  can  become  a  time 
consuming  exercise.  Notwithstanding  that  speech  consistency  is  the  panacea 
for  successful  operation,  it  may  pay  to  have  the  speaker  introduce  some 
variability  in  the  samples.  The  range  of  variability  must  be  traded  off 
against  the  number  of  samples  to  be  collected.  The  introduction  of 
environmental  variabi 1 ities  into  data  collection  may  be  advisable,  but  this 
procedure  has  similar  constraints  with  the  number  of  samples  collected. 

Speech  Recognition  Testing 

This  feature  enables  the  trainee  to  determine  that  the  recognition 
system  is  working  correctly.  Testing  should  be  executed  as  a  separate 
(sub)  mode  and  should  reflect  the  digits/words/ phrases  that  have  beer, 
sampled.  The  test,  mode  allows  the  trainee  to  speak  any  vocabulary  item 
(for  which  a  reference  pattern  has  been  established)  to  determine  that  it 
is  being  recognized  correctly.  The  test  mode  can  suggest  to  the  trainee 
the  vocabulary  items  that  the  recognition  system  is  having  trouble 
discriminating,  and  suggest  new  speech  samples  to  overcome  the  problem. 

Initiation  of  the  recognition  test  mode  can  be  made  by  the  system,  by 
the  trainee  or  by  the  instructor.  System  initiation  of  the  test  mode  can 
be  designed  to  occur  at  the  end  of  a  submode  when  recognition  accuracy 
declines  below  some  threshold. 

A  CRT  terminal  is  a  useful  medium  for  displaying  the  recognition 
response  to  the  test  utterances.  Speech  generation  is  an  alternative 
method  to  provide  feedback  information  about  the  recognized  item. 

Recognition  test  is  a  good  speech  "practice"  sub-mode.  It  allows  a 
new  trainee  to  build  confidence  that  he/she  is  speaking  in  a  consistent 
manner.  However,  unless  the  vocabulary  is  small,  say  under  20  words, 
random  test  utterances  by  the  user  should  give  away  to  a  more  orderly  set 
of  test  utterances  as  determined  by  the  speech  recognition  system.  A 
confusion  matrix  is  a  good  way  to  indicate  to  the  user  which 
digit/word/phrase  should  be  tested  (and  new  speech  data  collected).  The 
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confusion  matrix  relates  the  confusion  of  each  vocabulary  tern,  to  every 
other.  In  the  recognition  test  mode,  the  user  can  call  for  the  display  of 
the  total  matrix  or  only  those  utterances  which  have  a  high  confusion 
index.  A  confusion  index  provides  an  indicator  of  the  recognition  status 
of  the  vocabulary  items.  Based  on  the  confusion  index,  the  user  may  decide 
to  "retrain"  all  problem  items  before  continuing.  Alternatively,  he/she 
may  elect  to  "retrain"  only  selected  utterances  which  are  interfering  with 
the  progress  of  training. 

Speech  Recognition  Feedback.  Speed,  recognition  feeback  is  a  technique  of 
inmediate  verification  after  each  utterance  when  the  user  is  not  in  the 
test  mode.  However,  the  presentation  of  the  recognition  feedback  must  not 
interfere  with  the  primary  training  task.  Response  timing  for  recognition 
and  feedback  display  is  critical  in  a  fast  moving  scenario  (like  LSOTS).  A 
maximum  of  one  second  is  recommended  for  feedback  for  rapidly  paced  tasks. 
Design  decisions  with  respect  to  recognition  feedback  should  focus  on 
making  the  information  available  to  the  trainee  without  disruption  of  the 
training  task.  Some  variables  to  consider  include  type  of  presentation, 
visual  or  aural,  location  of  visual  presentation,  and  temporal 
characteristics  (constant  parameters  or  task-dependent). 

Normally  the  feedback  message  should  be  presented  passively  on  the 
primary  training  display.  This  is  easily  accomplished  when  the  display  is 
a  CRT.  However,  where  no  CRT  is  used,  an  alphanumeric  display  in  the  main 
field  of  viewing  activity  is  a  suitable  alternative. 

Recognition  "flagging"  is  a  technique  for  informing  the  user  that 
system  recognition  has  taken  place,  without  stating  what  actually  was 
recognized.  This  technique  is  not  recommended,  because  it  can  mislead  the 
user  when  the  utterance  has  been  recognized  incorrectly.  It  could  be  a 
useful  technique,  however,  in  systems  with  very  high  (99%)  recognition 
accuracy. 

A  situation  display  change  is  an  active  but  indirect  way  of  presenting 
user  feedback.  This  technique  is  good,  providing  that  the  situation  change 
is  consonant  with  correct  recognition,  and  that  the  time  required  for  the 
display  change  does  not  delay  the  pace  of  the  training  task.  If  the  delay 
is  too  long,  it  can  induce  the  user  to  make  another,  normally  more 
emphatic,  utterance  to  attempt  a  successful  recognition.  Unfortunately, 
more  emphasis  causes  recognition  errors  and  eventual  frustration  of  the 
user. 

SPEECH  SYSTEM  SPECIFICATION 

The  speech  system  specification  is  normally  divided  into  two  parts, 
recognition  and  generation.  The  former  is  discussed  herein.  No  attempt 
should  be  made  to  develop  a  speech  system  specification  unless: 

1.  A  list  of  system  requirements  has  been  developed  and  approved. 
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2.  The  match  between  system  requirements  and  projected  technology 
shows  that  production  of  the  system  is  feasible. 

3.  System  operating  and  human  factors  design  has  been  completed. 

4.  Speech  features  required  by  other  subystem,  (like  Instructor  Model 
and/or  Simulation  and  Event  Control)  are  properly  delineated. 

Many  specifications  are  not  particularly  useful  because  they  are 
written  generically  rather  than  specifically.  The  generic  sense  reflects 
that  the  writer  does  not  know  sufficient  detail  to  be  specific  at  the  time 
of  writing.  If  the  generic  specification  becomes  part  of  a  contract  then 
both  parties  are  endangered.  It  is  best  to  attempt  to  specify  everything 
and  to  identify  missing  details  as  "TBD"  -  to  be  determined. 

The  structure  of  the  speech  recognition  specification  should  be  based 
on  the  features  list  shown  in  Table  A-3  and  organized  hierarchically  as 
proposed  in  Figure  A-3.  The  document  describing  the  system  operating  and 
human  factors  design  should  be  included  as  an  appendix  to  the  specifi¬ 
cation.  In  order  for  the  specification  to  be  a  useful  document,  detailed 
descriptions  are  required  of  hew  the  speech  system  will  operate  and  its 
interface  with  the  trainee  and/or  the  human  instructors. 
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SPEECH 

CHARACTERISTICS 


-Continuous 
Speech 
Recognition 
-Connected 
Speech 
Recognition 
-Isolated  Word 
Recognition 
-Vocabulary 
Size 

-Subsetted 

Vocabularies 

-Speech 

Stylization 

Requirements 


SPEECH  SYSTEM 

REQUIREMENTS  SPECIFICATION 

SPEECH  GENERATING 
SYSTEM  (See  Figure  A-5) 


I 

functional 

DESIGN 

FEATURES 


-Voice  Data 
Collection 
-Speech  Recog¬ 
nition  Practice 
-Recognition  Test 
-Confusibi 1 ity 
Index 

-Repeated  Voice 
Data  Collection 
-Speech  Recog¬ 
nition  Feedback 
-Syntax  Control 
-Understanding 
-Multiple  Con¬ 
current  Users 
-Methods  for 
Signing  On 
-5peech  Recou- 
nition  On/Off 
Control 


1 

(RECOGNITION 

PERFORMANCE 

CHARACTERISTICS 


-Speaker  Dependency 
-Transmission  Recog¬ 
nition  Accuracy 
-Rejection  Error 
-Speaking  Rates 
-Pause  Duration 
-Reaction  Time 
-Reject  ion 
Thresholds 
-User  Variabil ities 
-Environmental 
Perturbations 


MISCELLANEOUS 

FEATURES 


-Projected  Cost 
-Speech  Input  Device 
-Input  Level  Control 
-Off  the  Shelf  Hardware 
-Software  Requirements 
-Central  Processing 
Requirements 
-Distributed  Processing 
Requirements 

-Outputs  to  Other  Systems 


Figure  A-3.  Speech  System  Specifications:  Recognition  System  Requirements 
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DESIGN  GUIDELINES 


GENERAL  APPROACH 

Figure  A-l  presents  a  design  procedure  intended  to  provide  guidance  to 
training/system  analysts  and  systems  engineers  (the  training  system  design 
team)  on  the  implementation  of  an  automated  speech  recognition  system  to 
support  training. 

DEVELOP  A  SPEECH  TRAINING  TASK  VOCABULARY 

Conduct  a  speech  task  and  vocabulary  analysis  based  on  the  output  of 
the  ISD  process. 

Identify  those  utterances  spoken  by  the  trainee  in  the  conduct  of 
training. 

Identify  those  utterances  spoken  by  the  instructor  (or  others)  to  the 
trainee  during  the  course  of  training. 

Develop  a  speech  task  vocabulary  which  is  operationally  acceptable  and 
technically  feasible. 

Identify  and  document  any  words  and  phrases  which  are  expected  to  be 
difficult  for  the  system  to  recognize  but  which  are  essential  to  training. 

Identify  any  speech  task  time  constraints  imposed  by  the  conduct  of 
the  training  scenario. 

IDENTIFYING  THE  SPEECH  RECOGNITION  SYSTEM  REQUIREMENTS 
Speech  Recognition  Characteristics 

Select  the  speech  recognition  characteristics  required  of  the  system 
in  accordance  with  Table  A-3  A.  Decide  on  the  type  of  recognition  system 
which  will  best  meet  the  training  needs:  continuous,  connected  or  isolated 
speech  recognition.  Decide  on  the  vocabulary  size  required.  To  allow  for 
unforeseen  expansion,  this  should  be  at  least  10%  larger  than  the 
vocabulary  size  determined  from  the  speech  task  and  vocabulary  analysis. 

Identify  how  the  vocabulary  can  be  subsetted  as  a  function  of  the 
operating  environment  to  enable  a  higher  order  processing  (understanding) 
of  words  and  phrases. 

Ensure  that  the  user  understands  the  stylization  constraints  imposed 
by  the  recognition  system.  In  particular,  pausing  constraints  should  be 
consistent  with  the  operational  language. 
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Functional  Design 

Develop  the  functional  design  of  the  recognition  system  in  accordance 
with  Table  A-3  B. 

For  speaker  dependent  systems,  decide  how  speech  data  collection  and 
recollection  will  be  implemented  (off-line,  special  data  collection 
scenario  in  the  training  context,  transparently  in  progressive  stages,  or  a 
combination  of  techniques). 

Identify  how  and  when  speech  recognition  practice  is  to  take  place, 
how  recognition  testing  will  be  implemented,  and  how  word  and  phrase 
recognition  confusion  data  will  be  presented  to  the  trainee  (and  human 
instructor). 

Decide  how  speech  recognition  feedback  information  will  be  presented 
to  the  trainee  (and  human  instructor).  Ensure  that  the  feedback  infor¬ 
mation  does  not  interfere  with  the  conduct  of  the  training  task. 

Develop  the  operational  rules  and  probabilities  which  govern  the  use 
of  the  task  vocabulary  so  that  syntax  control  logic  can  be  developed  to 
provide  a  higher  order  of  speech  understanding. 

If  the  training  system  is  to  be  designed  for  multiple  concurrent  users 
state  that  the  speech  recognition  response  for  each  trainee  station  will 
not  exceed  one  second  and  that  each  station  will  operate  independently  from 
all  other  stations.  If  there  is  any  doubt  about  the  statement,  the  systems 
engineer  should  request  a  thorough  review  of  the  multiple  user  concept. 

Develop  the  method  of  signing  on  for  the  trainee  and  human  instructor 
and  decide  on  the  use  of  peripheral  devices  such  as  diskettes  and 
high-density  magnetic  cards.  Aspects  to  be  considered  are  the  storage  of 
the  user's  speech  reference  patterns,  training  system  access  security,  and 
the  storage  of  trainee,  instructor  and  class  records.  These  aspects  should 
be  coordinated  with  the  training  system  administrators. 

Determine  the  optimum  method  of  keying  the  control  of  the  user  inputs 
to  the  recognition  system. 

Recognition  Performance  Characteristics 

Select  the  speech  recognition  system  performance  characterics  in 
accordance  with  Table  A-3  C. 

Determine  what  transmission  accuracy  and  rejection  error  is  acceptable 
for  the  system.  Everyone  would  like  to  see  100%  and  0%  respectively. 
However,  the  operational  community  can  be  expected  to  yield  to  figures 
slightly  less  than  perfect.  But,  it  is  vital  to  state  that  these  figures 
will  be  obtained  in  the  operational  environment  using  typical  trainees. 
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Current  recognizers  will  not  allow  a  vocabulary  item  to  exceed  2-5 
seconds  (depending  on  the  manufacturer).  Recognition  of  discrete  words  can 
be  expected  up  to  rates  of  50  words/minute,  and  possibly  as  high  as  180 
words/minute.  Current  isolated  word  recognition  systems  require  a  minimum 
pause  of  100  milliseconds  between  words  and  phrases.  Newer  techniques  are 
reducing  this  to  less  than  50  milliseconds.  The  recognition  system  should 
begin  to  respond  in  a  manner  perceptible  to  the  trainee  within  1  second  of 
utterance  completion. 

An  adjustable  rejection  threshold  is  recommended.  Rejection  levels 
should  be  adjusted  to  achieve  optimal  performance  within  the  context  of  the 
training  task.  The  opinions  of  the  user  community  should  be  elicited  to 
assist  in  rejection  threshold  adjustment. 

Major  sources  of  speech  variability  should  be  determined,  and 
appropriate  countermeasures  undertaken.  Short  term,  but  consistent  voice 
changes  (such  as  fatigue  or  a  cold)  can  be  countered  with  an  updated 
temporary  speech  data  collection  file.  Stress  effects  should  be  reduced  by 
any  effective  means.  Examples  are  to  create  stress  during  speech  data 
collection,  to  provide  training  on  maintaining  consistent  speech  despite 
stress,  or  to  reduce  the  stress. 

Environmental  perturbations  should  be  determined  and  eliminated 
whenever  possible.  Consistent  noise  may  be  countered  by  performing  speech 
data  collection  with  the  noise  present. 

Mi  seel  1 aneous  Features 


Develop  other  features  of  recognition  system  in  accordance  with  Table 
A-3  D. 


Be  realistic  about  system  cost.  Consider  whether  a  stand-alone 
recognizer  is  sufficient,  or  will  "understanding"  software  need  to  be 
developed. 

A  close-talking,  noise  cancelling  microphone  is  recommended  for  speech 
inputs  to  the  recognition  system.  Hand  held  devices  are  less  desirable 
than  a  headset  with  boom  microphone  because  they  add  another  source  of 
variability.  A  high  response,  automatic  gain  control  input  is  recommended 
to  enhance  consistency  of  speech  input. 
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REASSESS  THE  SPEECH  TECHNOLOGY  STATE-OF-THE-ART 

Speech  technology  is  advancing  rapidly.  The  training  systems  design 
team  should  reassess  the  state-of-the-art  to  determine  what  features  and 
capabilities  are  available.  These  capabilities  must  be  matched  to  the 
requirements  of  the  speech  recognition  system. 

A  list  of  major  features  and  capabilities  is  given  in  Table  A-3.  This 
list  should  be  used  to  aid  the  technology  reassessment.  Sources  of 
information  should  include  a  review  of  the  most  recent  recognition  systems 
applications  (outside  the  laboratory),  and,  with  reservations,  review  of 
the  latest  claims  made  by  the  manufacturers. 

MAKE  A  SPEECH  RECOGNITION  TECHNOLOGY  PROJECTION 

Make  the  technology  projection  based  on  six  key  points  discussed  in 
the  previous  section  under  the  subheading  "Making  a  Speech  Recognition 
Technology  Projection."  Do  not  be  concerned  (at  this  stage)  whether  the 
available  technology  does  not  match  the  system  requirements,  so  far  as  they 
have  been  defined  (see  "Matching  Technology  to  System  Requirements"). 

Systematically  match  the  technology  to  the  previously  developed  system 
requirements.  The  performance  of  some  technology  may  not  be  up  to 
expectations.  Therefore,  each  match  should  be  given  a  rating  or  weighting. 

If  the  technology  features  will  not  meet  system  requirements,  then  the 
system  requirements  may  have  to  be  scaled  down  and  the  matching  process 
reiterated.  Involve  the  operational  community  in  this  reiteration  process 
and  document  the  system  requi rements  agreed  upon.  Remember  you  are  looking 
for  a  yes/ no  matching  answer.  But  note  that  over  projection  of  the 
technology  can  produce  a  training  system  which  could  be  unusable;  whereas 
under  projection  will  produce  a  useable  system  that  will  not  necessarily 
fulfill  the  operational  requirement.  (The  system  designer  can  be  caught 
either  way!) 

SPEECH  SYSTEM  OPERATING  AND  HUMAN  FACTORS  DESIGN 

As  soon  as  the  decision  is  made  to  proceed  with  the  system  design, 
describe  in  detail  how  the  speech  recognition  and  generation  systems  will 
operate.  The  description  must  fulfill  the  needs  of  operational  training 
managers  as  well  as  software  programming  staff  so  that  no  operational 
ambiguities  exist. 
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Develop  a  step-by-step  operating  scenario  in  conjunction  with  a 
subject  matter  expert.  Describe  each  action  which  takes  place  in  the 
interface  between  the  training  device  and  the  trainee  in  as  much  detail  as 
possible. 

Expect  to  discover  a  series  of  questions  involving  human  factors 
design  and  operating  procedures  which  have  never  been  addressed  before. 
Try  to  resolve  these  questions  as  they  arise,  before  continuing  with  the 
remaining  scenario  development. 

Important  human  factors  considerations  for  the  speech  recognition 
system  center  on  speech  data  collection,  recognition  testing,  and 
recognition  feedback  (see  the  previous  section,  subheading  "Human  Factors 
Considerations"). 

Document  the  system  operating  and  human  factors  design  as  the 
preliminary  functional  design  description. 

DEVELOP  A  SPEECH  SYSTEM  REQUIREMENTS  SPECIFICATION 

Document  the  speech  recognition  (and  generation)  system  requirements 
in  a  specification.  The  system  specification  should  include  the 
preliminary  system  functional  design  document  as  an  attachment. 

No  attempt  should  be  made  to  develop  a  speech  system  specification 
unless: 

1.  A  list  of  system  requirements  has  been  agreed  upon. 

2.  The  match  between  system  requirements  and  projected  technology 
indicates  that  production  of  the  system  is  feasible. 

3.  The  system  operating  and  human  factors  design  has  been  completed. 

4.  The  speech  features  required  by  other  subsystems  are  properly 
del ineated. 

The  speech  recognition  requirement  specification  should  consider  the 
requirements  set  out  in  Figure  A-2. 
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PART  TWO 

SPEECH  GENERATION  SYSTEMS 


INTRODUCTION 

This  design  guide  is  intended  to  encompass  a  design  procedure  for 
speech  generation  (synthesis)  systems,  as  depicted  in  Figure  A-4.  The 
steps  in  the  procedure  are  as  follows: 

1.  Extend  the  ISO  process  to  include  speech  task  analysis  for  a 
planned  training  system  development  endeavor. 

2.  Identify  the  speech  generation  system  design  requirements  from  the 
task  analysis. 

3.  Update  a  technical  projection  of  the  automated  speech  technology 
"state-of-the-art"  to  facilitate  a  decision  about  whether  the 
design  requirements  for  the  speech  generation  system  can  be  met. 

4.  Make  the  technical  projection  whether  the  design  requirements  for 
the  speech  generation  system  can  be  met  in  the  time  frame  in  which 
the  training  system  is  required  to  start  operating. 

5.  Make  the  planning  decision  that  the  speech  generation  system 
features  can  be  met  within  the  technical  projection  for  a  specified 
production  system  time  frame. 

6.  Develop  the  speech  system  operating  and  human  factors  design  so 
that  the  operation  of  the  system  can  be  thoroughly  described  for 
all  personnel  involved  with  the  design  and  development  process. 

7.  Develop  the  speech  system  specification  so  that  it  becomes  an 
effective  document  for  the  designers,  developers  and  users  of  the 
training  system. 

The  use  of  the  words  "recording  and  playback"  in  the  foregoing  outline 
presupposes  that  all  communications  made  to  or  by  the  trainee  are  recorded 
for  each  training  session  and  are  available  to  be  played  back  as  required. 

SPEECH  TASK  AND  VOCABULARY  ANALYSIS 

The  speech  task/vocabulary  analysis  should  be  conducted  concurrently 
for  speech  generation  and  recognition.  It  Is  the  responsibility  of  the 
generation  system  to  provide  those  utterances  normally  made  by: 
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Note 

Work  l torn*  shown  in 
chained  blocks  should 
be  carried  out  con¬ 
currently  with  the 
equivalent  speech 
recognition  items 
(See  Figure  l  ) 
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PRELIMINARY  FUNCTIONAL 
DESCRIPTION 


Figure  A-4.  Work  Flow  for  Speech  Generation  System  Design 
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1.  The  automated  (psuedo)  instructor  in  the  conduct  of  training. 

2.  Other  persons  who  transmit  verbal  information  to  the  trainee  during 
the  conduct  of  the  tasks,  e.g.,  pilots,  air  traffic  controllers, 
tactical  coordinators,  team  members,  etc. 

3.  The  training  system  in  the  form  of  cues  and  prompts  which  are 

required  to  provide  instruction  to  the  trainee. 

Each  simulated  speaker  should  be  established  with  a  separate 
vocabulary  of  digits/words/phrases  to  be  generated.  Where  there  will  be 
substantial  reliance  on  the  spoken  word  from  the  training  system  to 

instruct  the  trainee,  a  basic  speech  generation  vocabulary  capacity  of  500 
words  is  recommended. 

TIME  CONSTRAINTS 

Time  constraints  for  speech  generated  messages  occur  when  there  are 
more  than  one  message  transmission  channel  available  and  when  there  are 
multiple  messages  to  be  transmitted  on  each  channel. 

Generated  messages  should  be  automatically  overridden  when  the  trainee 
is  speaking,  provided  that  such  overriding  is  in  context  with  the 
operational  environment.  Alternatively,  the  generated  message(s)  may  be 

allowed  to  continue  but  should  not  interfere  with  the  speech  recognition 

process. 

Multiple  messages  from  a  different  sources  can  be  transmitted  to  thr 
trainee  simultaneously,  provided  that  they  are  in  context  with  the 
operational  environment. 

If  more  than  one  message  is  to  be  transmitted  over  one  channel  to  the 
trainee,  the  messages  should  be  prioritized.  Once  each  message 
transmission  has  commenced,  it  should  not  be  overridden  by  a  higher 
prioritized  message  on  the  same  channel.  If  the  speech  generation 
channel (s)  can  be  overridden  by  the  trainee's  transmission,  then  the 
partially  transmitted  message  should  be  repeated  in  full,  provided  that 
such  repetition  is  in  context  with  the  training  scenario. 

Generated  speech  messages  which  are  used  to  cue  and  prompt  the  trainee 
for  instructional  purposes  should  not  be  repeated  more  than  twice. 

With  the  foregoing  time  constraints,  it  is  necessary  for  the 
training/system  analysts  to  develop  inter-  and  intra-  channel  priority 
schedules  for  all  speech  generated  message  vocabularies. 

IDENTIFYING  THE  SPEECH  GENERATION  SYSTEMS  REQUIREMENTS 

The  purpose  of  this  step  in  the  development  process  is  to  produce  a 
written  definition  of  the  requirements  for  speech  generation,  recording  and 
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playback.  Table  A-5  lists  the  features  which  should  be  considered  in 
developing  the  requirements.  The  table  has  been  subdivided  as  follows: 

Speech  Generation  -  Requirements  which  relate  to  speech  messages 
generated  by  the  training  system. 

Speech  Recording  and  Playback  -  Requirements  which  relate  to  the 
recording  and  playback  of  utterances  made  by  the  trainee,  the  human 
instructor  (when  present)  the  pseudo  instructor  and  communicate  with 
the  trainee  during  training  sessions. 

REASSESSING  THE  SPEECH  TECHNOLOGY  STATE-OF-THE-ART 

The  reassessment  of  technology  for  speech  recognition  and 
generation  should  be  coordinated.  Speech  generation  technology,  as  applied 
to  voice  interactive  training  systems,  has  advanced  to  the  point  where 
solid  state  devices  are  available  to  provide  274  words  of  naturally  spoken 
male  speech  with  integral  controls  processing  for  under  $100.00. 

Both  speech  generation  and  recording/playback  technology  must  be 
assessed  and  compared  to  the  training  system  requirements.  Both  are 
advancing  rapidly,  so  a  useful  technology  assessment  must  include 
projection  of  the  capabilities  over  the  time-frame  of  the  training  system 
development.  The  recording  technology  assessment  should  include  the 
processes  of  recording,  analog  to  digital  conversion,  digital  compression, 
and  storage  in  random  access  solid  state  or  disc-type  memories.  Access 
time  for  a  particular  part  of  a  recording  becomes  an  important  performance 
criterion. 

Each  feature  should  be  related  in  a  systematic  manner  to: 

1.  The  most  recent  experience  gained  in  the  field  with  speech 
generation  recording  and  playback. 

2.  The  lastest  claims  and  demonstrations  made  by  manufacturers  in 
regard  to  a  specific  generation  recording  and  playback  capability. 

For  the  foreseeable  future,  the  use  of  the  "analysis/synthesis" 
method  of  speech  generation  is  recommended  for  all  training  system 
applications  because  it  provides  more  natural  speech  and  speaker 
variability  than  the  "synthesis-by-rule"  mechanization  (see  Michael  is  and 
Wiggins,  1981). 

MAKING  A  SPEECH  GENERATION  TECHNOLOGY  PROJECTION 

Having  established  the  current  state-of-technology,  the  training 
system  design  team  should  have  no  difficulty  in  making  a  projection  of 
where  the  generation  and  playback  technology  is  likely  to  be  in  the  time 
frame  of  building  the  training  system. 
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TABLE  A-5.  SPEECH  GENERATION  SYSTEM  POTENTIAL  REQUIREMENTS 


SPEECH  GENERATION 

Number  of  channels  required 
Speaker  Identity 

Sex,  dialect,  speaker  differences,  naturalness 
Vocabul aries 

Number  required,  size,  single  digits,  coupled  digits,  words, 
phrases , 

Priorities 

By  channel,  by  message  type,  message  repetition  protocol 
Message  Format 

Prerecorded,  analysis/synthesis,  synthesis-by-rule 
Output 

Type  of  device,  power  required,  frequency  response 

SPEECH  RECORDING  AND  PLAYBACK 

Number  of  speech  channels  to  be  recorded 

Type  of  recording 

Analog,  digital , 

Storage  Medium 

Tape,  disc,  random  access  memory  device 
Recording  control 

Continuous,  switch  activated,  voice  activated 
Extent  of  Recording 

One  training  session,  "N"  training  sessions 
Playback  control 

Indexing  by  time,  indexing  by  event,  in  combination 


AD-A129  145  VOICE  TECHNOLOGY  DESIGN  GUIDES  FOR  NAVY  TRAINING  1/1 

SYSTEMS(U)  CANYON  RESEARCH  GROUP  INC  WESTLAKE  VILLAGE  *  * 
CA  d  C  COTTON  ET  AL .  MAR  83  CRG-TR-82-006 
UNCLASSIFIED  NAVTRAEQUIPC-80C-0057- 1  N6 1339-80-C-0057  F/G  5/5  Nl 


MICROCOPY  RESOLUTION  TEST  CHART 

NATIONAL  BUREAU  Of  STANDARDS-196U  A 


NAVTRAEQUIPCEN  R0-C-0057-1 


There  are  at  least  six  key  points  on  which  the  speech  generation 
projection  should  be  made: 

1.  Ultimate  size  of  the  vocabulary. 

2.  Availability  of  multiple  voices. 

3.  Naturalness  of  each  voice. 

4.  Unit  price. 

5.  Reliability  (MTBF)  of  integral  parts. 

6.  Internal  or  external  processing  and  control. 


The  speech  recording  and  playback  projection  should  include: 

1.  Ultimate  storage  capacity  for  four  letter  words. 

2.  Storage  medium  -  integral  or  external. 

3.  Analog  or  digital  format. 

4.  Access  time  to  any  part  of  the  recording. 

5.  Unit  price. 

6.  Reliability  (MTBF)  of  integral  parts. 


MATCHING  TECHNOLOGY  TO  SYSTEM  REQUIREMENTS 

Based  on  the  definition  of  the  speech  generation  system  requirements 
and  the  technology  projection,  it  will  be  possible  to  make  a  match  between 
them.  If  there  is  not  a  complete  match  on  the  first  time  through,  it  is 
most  likely  than  some  factors  in  the  system  requirement  have  been 
overstated  or  alternatively,  the  technology  has  been  under  assessed.  In 
the  near  future  a  substantial  reduction  is  likely  in  the  cost  for  solid- 
state,  digitally  formatted  word  storage  devices.  Increased  storage 
capacity  is  expected  during  the  same  time  frame.  This  advancement  will  be 
the  natural  outcome  of  combining  micro  processors  and  cheap  bulk  memory 
onto  one  or  two  chips.  This  capability,  in  turn,  will  lead  to  distributed 
processing  for  the  various  functional  modules  of  speech-interacti ve 
training  systems  of  the  future. 

SPEECH  GENERATION  SYSTEM  OPERATING  AND  HUMAN  FACTORS  DESIGN 

Once  the  decision  to  proceed  with  system  design  has  been  made,  it  is 
important  for  the  training/system  analysts  to  describe  how  the  generating 
and  playback  system  will  operate.  This  description  should  be  developed  in 
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concert  with  the  recognition  system  functional  description,  described  in 
Part  1,  to  fulfill  the  needs  of  all  personnel  involved  with  the  training 
system  design  and  development. 

Training  Scenario  Development 

See  the  discussion  in  Part  1. 

Human  Factors  Considerations 


There  are  five  human  factors  considerations  for  speech  generation 
systems  that  are  common  to  all  speech  interactive  training  systems,  and 
should  be  considered  carefully  by  the  training/ systems  analyst. 

1.  The  naturalness  of  the  generated  speech. 

2.  The  timing  of  the  generated  output  always  should  be  in  context  with 
the  training  scenario. 

3.  The  generated  speech  response  should  start  within  a  very  short  time 
of  the  end  of  a  verbal  request  from  the  trainee  (nominally,  one 
second) . 

4.  For  playback,  the  access  to  a  recorded  section  of  speech  should  be 
fast  and  in  sync  with  the  training  scenario  being  represented. 

5.  When  multiple  messages  are  being  transmitted  over  one  channel,  the 
messages  are  prioritized  correctly  and  each  message  is  transmitted 
in  its  entirety. 

SPEECH  SYSTEM  SPECIFICATION 

The  speech  system  specification  is  normally  divided  into  tv/o  parts. 
Recognition  and  Generation.  The  former  is  discussed  in  Part  1  and  the 
latter  (which  includes  speech  recording  and  playback)  is  discussed  herein. 

No  attempt  should  be  made  to  develop  a  speech  system  specification 
unless: 

1.  A  list  of  desired  requirements  have  been  developed  and  approved. 

2.  The  match  between  system  requirements  and  the  technology  projection 
shows  that  production  of  the  system  is  feasible. 

3.  The  system  operating  and  human  factors  design  has  been  completed. 

4.  The  features  required  in  the  other  subsystems  ( i . e. ,  Instructor 
Model  and/or  the  Simulation  and  Event  Control  Model)  are  properly 
del ineated. 
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The  specification  for  speech  generation,  recording,  and  playback 
should  be  as  specific  as  possible.  This  is  feasible  because  the  speech 
generating  system  requirements  (identified  in  Table  A-5),  are  not  complex 
and,  furthermore,  the  current  technology  can  fulfill  nearly  any 
requirement  which  will  arise  in  the  near  future.  The  structure  of  the 
speech  generation  specification  should  be  based  on  the  requirements  list 
shown  in  Figure  A-5.  The  document  that  describes  the  system  operating  and 
human  factors  design  should  be  included  as  an  attachment  to  the  specifi¬ 
cation.  The  information  about  how  the  speech  generation  system  will 
operate  and  how  it  interfaces  with  the  trainee  and/or  human  instructor  must 
be  very  detailed  for  the  specification  to  be  a  useful  document. 
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DESIGN  GUIDELINES 


GENERAL  APPROACH 

Develop  a  speech  generation  system  design  in  accordance  with  the 
procedure  shown  in  Figure  A-4. 

DEFINE  THE  SPEECH  GENERATION  SYSTEM  REQUIREMENT 

Use  the  checklist  in  Table  A-5  to  develop  the  generation,  recording 
and  playback  system  requirements. 


REASSESS  THE  SPEECH  TECHNOLOGY  STATE-OF-THE-ART 

For  speech  generation  recording  and  playback,  the  reassessment  for  a 
specific  training  system  should  be  made  concurrently  with  that  for  speech 
recognition.  Because  the  former  technology  is  advancing  so  quickly, 
interim  reassessment  should  be  made  to  keep  abreast  of  development. 

Make  the  assessment  based  on:  I)  the  most  recent  experience  gained  in 
the  field  with  speech  generation,  recording  and  playback;  and  2)  the  latest 
claims  and  demonstrations  by  manufacturers  for  specific  products. 

DEFINE  THE  MESSAGE  TIME  CONSTRAINTS 

Define  the  number  of  message  transmission  channels  required  between 
the  generation  system  and  the  trainee  (and  human  instructor). 

Develop  the  inter-  and  intra-  channel  transmission  priority  protocol. 

Define  the  inter-  and  intra-  channel  transmission  priority  for  each 
word  and  phrase  group. 

GENERATE  SPEECH  VOCABULARIES  ANALYSIS 

Define  the  speech  generation  vocabularies  to  be  used  by  the  various 
people  whose  voices  are  simulated  by  the  system.  Examples  are  an  automated 
instructor,  pilots,  air  traffic  controllers,  tactical  controllers, 
simulated  team  members,  etc. 

Determine  the  number  of  different  words  required  by  each  vocabulary. 
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DESIGN  GUIDELINES 


MAKE  A  SPEECH  GENERATION  TECHNOLOGY  PROJECTION 

Make  the  projection  based  on  the  twelve  key  points  discussed  in  the 
previous  section,  under  the  heading  "Making  a  Speech  Generation  Technology 
Projection." 

Ensure  that  integral  or  external  processing  and  control  software  is 
available  to  fulfill  the  message  protocol  and  prioritization  requirements. 

For  recording,  trade  off  the  cost  benefits  of  an  integral  storage 
approach  versus  the  use  of  an  external  device  (like  an  analog  tape 
recorder).  If  high  speed  playback  access  is  not  a  training  requirement, 
then  a  high  quality  reliable  tape  recorder  may  be  cost  effective. 

MATCH  THE  SYSTEM  REQUIREMENTS  AND  THE  TECHNOLOGY  PROJECTION 

Repeat  the  process  until  the  requirements  match  the  technology 
available.  Be  sure  that  changes  made  to  the  requirements  are  acceptable  to 
the  operational  community  and  their  expectations  of  the  speech  generation 
system. 

Do  not  concentrate  on  the  speech  recognition  matching  effort  to  the 
detriment  of  speech  generation.  The  latter  is  just  as  important  because  it 
is  an  essential  link  between  the  machine  and  the  man. 

Make  a  Go/No  Go  decision  to  continue  with  the  speech  generating  system 
design  based  on  the  outcome  of  this  matching  process. 

DEVELOP  THE  SPEECH  SYSTEM  OPERATING  AND  HUMAN  FACTORS  DESIGN 

Develop  the  speech  system  operating  and  human  factors  design  for 
speech  generation,  recording,  and  playback.  This  design  task  should  be 
accomplished  in  concert  with  the  design  for  the  speech  recognition  system. 

The  training  scenario  should  be  developed  in  as  much  detail  as 
possible.  All  personnel ,  from  software  programmers  to  training  managers, 
should  be  fully  informed  about  what  the  training  system  will  do  in  each  of 
it's  operating  modes.  Unanticipated  problems  should  be  dealt  with 
immediately.  Unsolved  problems  could  be  disastrous,  should  they  be 
neglected. 

Ensure  that  the  five  human  factors  considerations  discussed  in  the 
previous  section  are  acted  upon. 

Document  the  interaction  between  the  training  scenario  and  the  system 
operating  and  human  factors  design  as  the  preliminary  functional  design 
description  for  the  speech  system. 
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DESIGN  GUIDELINES 


DEVELOP  THE  SPEECH  SYSTEM  REQUIREMENT  SPECIFICATION 

Part  1  of  this  specification  covers  the  speech  recognition  system  and 
Part  2  the  speech  generation  system. 

Both  parts  of  the  speech  system  specification  should  be  developed  only 
when  the  following  steps  have  been  completed:  a  list  of  functional 
requirements  has  been  developed;  there  is  a  match  between  the  requirements 
and  the  technology  projection;  the  system  operating  and  human  factors 
design  has  been  completed;  and  the  speech  requirements  of  other  training 
subsystems  (Instructor  Model,  Simulation,  and  Event  Control,  etc.)  have 
been  properly  delineated. 

The  content  of  the  speech  generation  specification  should  cover  the 
system  requirements  set  out  in  Figure  A- 4. 


A  generic  type  of  specification  is  to  be  avoided.  Be  specific  and  do 
not  hesitate  to  use  "T8D"  (to  be  determined)  if  the  information  is 
unavailable. 
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APPENDIX  B 

AUTOMATED  INSTRUCTOR  MODEL  DESIGN  GUIDE 


105 


NAVTRAEQUIPCEN  80-C-0057-1 


TABLE  OF  CONTENTS 

Pa^e 

INTRODUCTION .  108 

DEFINING  INSTRUCTOR  MODEL  AUTOMATION .  110 

DEVELOPING  THE  SYSTEM  DESIGN  STRATEGY .  110 

DEVELOPING  THE  CURRICULUM .  112 

DEFINING  THE  PERFORMANCE  MEASUREMENT  AND 

EVALUATION  CRITERIA .  116 

DEVELOPING  THE  DIAGNOSTIC  SCHEMA  AND  MODEL 

OF  THE  TRAINEE .  11 9 

DEVELOPING  THE  INSTRUCTOR  MODEL  DESIGN 

REQUIREMENTS .  120 

Instructional  Requirements .  IPO 

Operating  Requirements .  120 

Physical  Requirements .  121 

IMPLEMENTING  AN  INSTRUCTOR  MODEL  TECHNOLOGY 

AND  UPDATE  PROJECTION .  126 

DECIDING  TO  PROCEED  WITH  THE  DESIGN .  127 

DEVELOPING  THE  INSTRUCTOR  MODEL  OPERATING  AND 

HUMAN  FACTORS  DESIGN .  127 

DEVELOPING  THE  INSTRUCTOR  MODEL  REQUIREMENT 
SPECIFICATION .  128 

DESIGN  GUIDELINES .  130 

LIST  OF  ILLUSTRATIONS 

Figure  B-l.  Automated  Instructor  Model 

Design  Guidel  ines .  109 

Figure  B-2.  Typical  Instructor  Model 

Top  Level  System  Block  Diagram...  113 


106 


NAVTRAEQUIPCCN  80-C-0057-1 


Figure  B-c 

Figure  B-* 

Table  B-l 
Table  B-2 
Table  B-3 

Table  B-4 


TABLE  OF  CONTENTS 
(Continued) 


Page 


.  Scheme  for  Delineating 
Performance  Criteria  for 
LSO  Perception  Training. 


.  Instructor  Model  Requirement 

Specification  Organization .  129 


LIST  OF  TABLES 


Important  Considerations  For 
Automated  Instructor  Models. 

Top  Level  System  Information 
Flow  Organization . 

A  Framework  For  Developing 
Instructor  Model  Designs.... 

Instructor  Model  Potential 
Design  Requirements . 


Ill 

114 

115 

122 


107 


NAVTRAEQUIPCEN  80-C-00b/-l 


AUTOMATED  INSTRUCTOR  MODEL  DESIGN  GUIDE 


INTRODUCTION 

The  design  guidelines  for  the  automated  instructor  model  are 
principally  oriented  toward  an  instructor  model  that  will  operate  with 
an  automated  speech  system  (see  Appendix  A).  However,  this  may  not 
always  be  the  case.  Where  departure  from  the  guidelines  herein  is 
necessary  for  a  system  without  automated  speech  technology,  the  document 
is  appropriately  annotated. 

The  design  guide  assumes  that  the  user  has  a  working  knowledge  of 
instructor  model  design  and  a  detailed  knowledge  of  the  training  task  to 
which  it  will  be  applied. 

Note  that  an  automated  instructor  model  does  not  preclude  the 
existence  of  a  human  instructor  station  as  part  of  the  traininq  system 
design. 

These  design  guidelines  are  intended  to  encompass  the  following 
design  procedures  as  depicted  in  Figure  B-l. 

1.  Define  the  degree  of  automation  suitable  for  the  instructor  model. 

2.  Develop  the  system  design  strategy  which  is  to  be  used  for 
accomplishing  the  training. 

3.  Develop  the  curriculum  which  will  fulfill  the  training  strategy. 

4.  Define  the  performance  and  evaluation  criteria  to  be  used  to 
measure  the  performance  of  the  trainee  throughout  the  conduct  or 
training  on  the  training  system. 

5.  Develop  the  diagnostic  schema  and  model  of  the  trainee(s). 

6.  Develop  the  instructional  operating  and  physical  design 
requirements  of  the  instructor  model. 

7.  Update  what  is  known  about  the  instructor  model  technology 
state-of-the-art  and  projecting  it  into  the  production  time  frame 
of  the  training  system. 

8.  Make  the  technical  decision  that  the  instructor  model  can  be  met 
within  the  required  time  frame. 

9.  Develop  the  instructor  model  operating  and  human  factors  design 
including  it's  interface  with  the  human  instructor. 


108 


TRAINING  SYSTEM 
IMPLEMENTATION  “ 
SCHEDULE 


INSTRUCTOR 

MODEL 

FEATURES  LIST 


Figure  B-l. 


NAVTRAEQUIPCEN  80-C-0057-1 


INSTRUCTOR 
MODEL 
FEATURES  LIST 


AUTOMATED 
INSTRUCTOR 
MODEL  DESIGN 
GUIDELINES 


Automated  Instructor  Model  Design  Guidelines 


109 


NAVTRAEQU I PCEN  80-C-0057-1 


10.  Dove) up  the  instructor  model  specification  so  that  it  becomes  an 
effective  document  for  the  designers,  developers,  and  users  of  the 
training  system. 

In  using  these  design  procedures  the  training/systems  analysts  and 
systems  engineers  should  be  cognizant  that  no  training  system  is  effective 
unless  it  provides  accurate  and  rapid  feedback  to  the  trainee  as  to  his  or 
her  performance. 

1)1  FINING  INSTRUCTOR  MODEL  AUTOMATION 

Before  the  functional  characteristics  of  an  instructor  model  can  be 
defined,  the  training  system  designers  must  decide  on  the  optimum  extent  of 
instructor  automation.  This  decision  can  be  considered  as  a  point  on  a 
dimension  ranging  from  no  automation  to  a  totally  instructorless  system. 
The  degree  of  automation  involves  many  considerations  the  more  important  of 
which  are  shown  in  Table  B-l.  Middle-level  automation  is  described  as  an 
instructor  support  system,  in  which  the  instructor  still  participates 
actively  in  every  training  or  event,  but  some  features  may  be  automated  to 
reduce  his  workload. 

Each  of  these  functions  may  be  partially  or  totally  automated. 
Decisions  about  the  degree  of  instructor  automation  should  be  based  on 
several  factors,  including: 

o  Manpower  availability 

o  Manpower  costs 

o  Instructor  model  development  cost 

o  Relative  training  effectiveness 

o  User  acceptance  and  utilization 

The  desired  level  of  instructor  automation  must  be  defined  early  in 
the  training  system  design  process  so  that  it  can  serve  as  a  guideline  for 
the  development  of  the  instructor  model.  Decisions  about  the  level  of 
instructor  automation  also  will  have  an  impact  on  the  design  of  other 
subsystems,  such  as  voice  and  simulation/event  control.  Therefore, 
decisions  about  instructor  automation  must  be  based  on  careful  analysis  of 
cost,  training  effectiveness,  and  user  acceptance. 

DEVELOPING  THE  SYSTEM  DESIGN  STRATEGY 

This  development  is  intended  to  delineate  what  is  required  of  the 
Instructor  Model  to  accomplish  a  specified  training  course.  It  is  a 
multi-faceted  problem  which  normally  can  be  solved  at  a  global  level. 
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TABLE  B-l.  IMPORTANT  CONSIDERATIONS  FOR  AUTOMATED 
INSTRUCTOR  MODELS 


TUTORIAL  PROCEDURES 

Presentation  of  Instructional  Materials 

General  Instructional  Approach 

Use  of  Special  Instructional  Features 

PERFORMANCE  ASSESSMENT 

Measurement 

Scoring 

Evaluation 

Diagnostics 

PERFORMANCE  FEEDBACK 

Knowledge  of  Results 

Error  Description 

Error  Reduction  Strategies 

CURRICULUM  CONTROL  AND  ADAPTIVE  LOGIC 

Remediation 

Advancement 

Adjust  Problem  Difficulty 
Problem  Initialization 
Curriculum  Branching 

RECORD  KEEPING 

Progress  in  Curriculum 
Detailed  Performance  Evaluations 
Performance  Summaries 
Diagnostics  Record 

TRAINEE  MODEL 


Learning  Models 
Optimization  Techniques 
Resource  Allocation 
Validation 
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However,  in  some  circumstances  it  will  require  a  prior  detailed  knowledge 
of  interfacing  equipment  before  any  strategy  can  be  addressed. 

The  degree  of  simplicity  (or  sophistication)  of  the  Instructor  Model 
is  highly  dependent  on  the  operational  community's  view  of  what  is  required 
of  the  overall  training  system.  For  instance,  if  the  operational  task  to 
be  trained  is  like  air  intercept  control,  the  instructor  model  may  be 
required  to  provide  automated  instruction,  curriculum  control,  performance 
measurement  and  evaluation,  instruction,  practice  exercises  and  record 
keeping.  On  the  other  hand,  if  the  instructor  model  is  to  be  added  to  an 
existing  trainer,  say,  for  night  carrier  landing  practice,  automation  may 
only  be  necessary  to  provide  briefing  instructions  for  a  series  of  canned 
approaches  and  performance  measurement  of  the  pilot. 

The  strategy  is  developed  by  analyzing  the  operational  training 
requirements  and  trying  to  arrive  at  a  top  level  system  solution.  A  good 
way  to  express  the  system  solution  is  in  a  block  diagram.  An  example  is 
shown  in  Figure  B -2.  Each  interconnecting  line  between  blocks  should  be 
identified  with  the  information  expected  to  flow  along  it.  An  organization 
for  this  task  is  shown  in  Table  B-2. 

In  those  cases  where  the  instructor  model  will  interface  with  existing 
equipment,  like  an  operational  flight  trainer,  the  training/systems 
analysts  will  need  the  services  of  a  simulator  design  engineer  to  determine 
the  input/output  information  flow  available  to  the  instructor  model.  This 
step  must  be  accomplished  before  the  instructor  model  design  strategy  can 
be  completed  (and  in  some  cases,  before  making  the  decision  whether  the 
operational  training  requirements  can  be  met  at  all). 

To  assist  in  the  development  strategy,  each  training  requirement, 
should  be  analyzed  from  the  aspects  of: 

1.  Modes  required  to  fulfill  each  requirement. 

2.  The  interfaces  required  with  the  trainee  (and  the  human  instructor) 
to  provide  the  flow  of  training  information. 

A  framework  for  conducting  the  the  analysis  is  shown  in  Table  B-3. 
DEVELOPING  THE  CURRICULUM 

Curriculum  development  is  a  critical  step  in  the  development  of  an 
effective  training  system.  Experts  in  training,  psychology,  and/or 
instructional  technology  should  be  tasked  with  curriculum  organization  and 
development.  It  is  beyond  the  scope  of  the  present  report  to  discuss  this 
process  in  detail.  Generally,  the  ISO  process  should  be  followed  to  define 
the  curriculum  organi zation ,  course  syllabi,  and  behavioral  objectives 
(Branson  et  al ,  1975;  Funaro  and  Mulligan,  1978;  Mulligan  and  Funaro,  1979; 
and  MIL-T-29053A(TD),  1979). 
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TABLE  B-2.  TOP  LEVEL  SYSTEM  INFORMATION 
FLOW  ORGANIZATION 
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TABLE  B-3.  A  FRAMEWORK  FOR  DEVELOPING  INSTRUCTOR  MODEL  DESIGNS 


MODES 

Signing  on  (and  off) 

Accessing  Records 
Selection  of  Training 
Review  of  Previous  Training 
Instruction  (or  teaching) 

Exercise  Practice 
Playback  of  Exercises 
Debriefing 
INTERFACES 

Audibly  with  the  trainee 

with  the  human  instructor  (when  present) 
Visually  with  the  trainee 

with  the  human  instructor 
Manually  with  the  trainee 

with  the  human  instructor 
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Strong  interaction  with  the  user  community  is  necessary  throughout  the 
curriculum  development.  A  subject  matter  expert,  ideally,  should  be  an 
integral  part  of  the  process.  The  curriculum  should  be  partitioned  into 
manageable  units  to  facilitate  development  of  the  courseware.  The 
partitioning  also  will  enable  subsequent  changes  in  the  curriculum  to  be 
managed  without  undue  time  and  cost. 

The  flexibility  of  the  system  to  change  with  changing  fleet 
requirements  should  be  a  system  design  goal  from  the  outset.  The  system 
design  team  must  plan  for  change  in  the  curriculum  or  the  large  investments 
in  software/coursewa re  wi  1 1  render  the  system  condemned  to  obsolescence 
before  it  is  delivered.  A  modular  approach  is  highly  recommended  for 
curriculum  development  and  its  eventual  representation  in  courseware. 

DEFINING  THE  PERFORMANCE  MEASUREMENT  AND  EVALUATION  (PM&E)  CRITERIA 

Criterion-based  training  performance  is  fundamental  to  the  design  of 
instructor  models.  In  an  adaptive  system,  the  measurement  and  evaluation 
of  trainee  performance  can  be  used  to  control  several  aspects  of  training 
including  position  in  the  syllabus,  level  of  difficulty,  and  time  in 
training. 

It  must  be  stressed  that  the  PM&E  criteria  must  be  defined 
explicity  early  in  the  design  process.  Failure  to  do  so  may  produce  later 
design  deficiencies  which  could  be  disastrous  to  the  eventual  success  of 
the  program. 

For  voice  interactive  training  systems,  speech  recognition  adds 
another  dimension  to  performance  measurement  which  the  training/systems 
analysts  must  consider.  In  this  case,  utterances  from  the  trainee  contain 
measureable  data.  Consider  the  expression  "turn  left  heading  270."  The 
words  "left"  and  "270"  are  data  on  which  the  air  traffic  control  In 
performance  can  be  measured.  Failure  of  the  speech  system  to  recognize 
properly  makes  it  impossible  for  the  performance  measurement  to  be 
accurate,  irrespective  of  how  well  the  performance  criteria  have  been 
defined. 

Automation  of  the  instructor  function  requires  that  PM&E  criteria  be 
quantified.  In  most  training  situations  the  human  instructor  uses  a 
mixture  of  subjective  and  objective  measures  to  evaluate  a  trainee.  For 
this  design  task,  each  subjective  measure  should  be  documented  at  each 
stage  of  training  and  translated  into  quantifiable  information. 
Furthermore,  objective  data  already  used  by  human  instructors  should  be 
scrutinized  for  source  and  accuracy  before  it  is  accepted  as  bonafied 
quantative  information. 

If  the  training  and  systems  analysts  cannot  accomplish  this  step,  the 
efficiency  of  using  an  instructor  model  for  PM&E  should  be  critically 
questioned.  It  is  better  to  state  that  the  human  instructor  could  do  a 
better  job  than  the  instructor  model  PM&E  design  can  be  expected  to 
provide. 
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One  of  the  principles  of  ISO  is  that  it  is  directed  towards  criterion 
performance  based  training.  However,  such  information  is  often  too  global 
and  insufficiently  systematic  to  lead  to  the  generation  of  an  instructor 
model  software  program. 

For  a  successful  PM&E  system  to  emerge,  the  curriculum  already  must  be 
described  in  detail  so  that  the  satisfactory  performance  for  each  key 
concept  and  training  objective  can  be  quantified. 

An  example  of  the  PM&E  analysis  process  is  depicted  in  Figure  B-3. 
Perception  of  the  aircraft  position  in  space  (line-up  and  glide  slope)  is  a 
key  concept  to  be  mastered  in  ISO  training.  Training  for  the  development 
of  line-up  perception  involves  estimation  of  lateral  position  and  rate  by 
the  LSO.  For  lateral  position,  displacement  criteria  from  flight  path 
centerline  need  to  be  established  for  the  LSO  calls  of: 

"Check  Your  Line-Up"  (+_  1/2°  from  centerline) 

"Little  Left/Right  For  Line-Up"  (_+  1°  from  centerline) 

“Right/Left  For  Line-Up"  (+_  2°  from  centerline) 


(These  are  examples  only.  See  McCauley  and  Borden,  in  press,  for  detail  of 
LSO  model ing). 

Establishing  the  criteria  often  will  require  multiple  opinions  of 
subject  matter  experts.  Variability  in  these  opinions  is  to  be  expected. 
Some  type  of  consensus  should  be  sought,  but,  in  some  instances,  it  is 
necessary  to  collect  data  in  the  operational  environment  or  analyze  actual 
operational  data  to  arrive  at  the  criteria.  Both  can  be  costly. 

In  some  circumstances,  quantative  data  are  available  in  discrete  form 
(0  or  1)  that  the  trainee  accomplishes  a  specific  task,  e.g.,  lowering 
landing  gear.  However,  while  the  discrete  event  is  very  amenable  for  PM&E, 
other  circumstances  must  be  considered  which  describe  the  timing  of  the 
event,  like  landing  gear  down  at  <  1500  &  >  100  foot  altitude. 

When  the  quantification  process  has  been  completed  the  extent  of  the 
measurement  task  should  be  assessed  and  documented.  The  training  and 
systems  analysts  should  decide  on  a  measurement  philosophy  that  is 
commensurate  with  the  size  of  the  task  and  the  computing  resources  that  are 
likely  to  be  available. 

Two  measurement  methods  prevail,  "collect  everything"  and  "measure  by 
rules."  The  first  method  is  to  measure  and  record  all  continuous  variables 
and  events  throughout  each  training  session  and  select  only  those  measures 
which  are  specifically  needed  for  a  particular  lesson,  event,  or  sequence. 
This  method  provides  most  flexibility  for  subsequent  modification,  if 
measurement  difficulties  are  encountered  in  the  field.  However,  the 
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Figure  B-3.  Scheme  for  Delineating  Performance 
Criteria  for  LSO  Perception  Training 
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requirements  for  temporary  storage  of  continuously  variable  data  can  be 
large.  The  second  method  is  to  measure  only  those  variables  and  events 
which  are  specified  for  a  particular  lesson,  event  or  sequence.  The 
logical  control  for  this  method,  however,  is  more  complicated. 

The  "collect  everything"  method  is  most  suited  to  application  where 
there  are  only  a  small  number  of  continuous  variables  to  be  measured; 
and/or  there  is  measurement  uncertainty  at  the  design  stage  which  will  have 
to  be  overcome  later  by  field  experience.  The  "measure  by  rules"  is  most 
suited  to  measuring  a  large  number  of  variables  at  specifically  defineable 
events  or  times,  and  when  the  rules  can  be  established  early  in  the  design 
process. 

A  model  to  convert  the  various  performance  measurements  into  an 
evaulation  score  of  the  trainee  should  be  developed  concurrently  with 
developing  the  actual  performance  criteria.  The  model  must  be  developed 
through  the  valuable  services  of  experienced  instructors.  Normally  the 
analysts  can  develop  a  simple  algorithm  that  can  be  used  to  demonstrate 
scoring  to  the  instructors.  However,  instructors  tend  to  make  allowances 
for  subjective  variables  like  "good  attitude."  The  inability  of  automated 
measurement  to  account  for  such  variables  may  precipitate  adverse  criticism 
of  the  scoring  algorithm.  There  is  no  guideline  to  accommodate  this  type 
of  subjectivity  in  automated  performance  measurement. 

DEVELOPING  THE  DIAGNOSTIC  SCHEMA  AND  MODEL  OF  THE  TRAINEE 

Diagnosis  of  the  trainee's  performance  over  a  series  of  training 
sessions  is  intended  to  determine  when  remediation  is  required  for  the 
trainee.  This  can  be  accomplished  by  adapting  the  syllabus  to  fulfill  the 
remediation  needs,  or,  if  the  syllabus  is  completely  linear,  by  providing 
special  training  off  to  the  side  of  the  curriculum  flow.  Both  techniques 
require  knowledge  of  how  the  typical  trainee  performs  at  each  point  of 
progression  through  the  training  course. 

The  use  of  trainee  models  and  diagnostic  models  are  sometimes 
considered  by  the  uninformed  to  be  of  dubious  value  because  they  are 
difficult  to  program  and  their  effectiveness  is  largely  unproven.  However, 
they  have  the  potential  of  providing  the  highest  efficiency  for  automated 
training.  To  achieve  this  efficiency,  continuous  in-service  expansion  of 
the  diagnostics  and  trainee  models  must  be  provided  before  any  benefits  can 
be  obtained. 

Provision  can  be  made  during  the  design  so  that  both  models  will 
continuously  adapt  by  "experience"  and  optimize  themselves. 

Optimization  techniques  for  automated  adaptive  training  systems  have  been 
described  by  Chatfield  and  Gidcumb  (1977).  Trainee  models  and  optimization 
techniques  for  adaptive  training  can  be  considered  a  form  of  artifical 
intelligence  (AI).  The  application  of  AI  techniques  for  voice-based 


119 


NAVTRAEQU1PCEN  80-C-0057-1 


training  systems  is  currently  under  investigation  by  NAVTRAEQU1 PCEN,  and 
one  study  in  the  program  has  been  completed  recently  (Chatfield,  Klein,  and 
Coons,  1981). 

Diagnostic  and  trainee  models  should  be  included  from  the  inception  of 
training  system  design,  irrespective  of  how  rudimentary  these  models  may 
be.  Initially,  the  output  of  the  models  should  neither  alter  the 
curriculum  followed  by  the  trainee  nor  provide  diagnostics.  Only  qualified 
instructors  and  analysts  would  have  access  to  the  information  to  support 
further  development  of  the  models. 

In  principle,  the  initial  diagnostic  model  should  be  predicated  on  an 
expansion  of  the  scheme  shown  in  Figure  B-3.  For  example,  the  diagnostic 
model  for  the  LSOTS  should  be  designed  to  consider  repeated  instances  where 
either  line-up  or  glide-slope  performance  is  mediocre  at  ranges  beyond  1/2 
mile.  Initially  this  might  be  diagnosed  as  "poor  distance  perception." 
The  diagnostic  would  automatically  look  at  whether  the  mediocre  performance 
was  related  to  day  or  night  approaches,  by  type  of  airplane,  whether  the 
perception  difficulty  had  a  "lateral  or  vertical  flight  path  preponderance, 
and  make  remedial  recommendations  accordingly. 

In  principle,  the  initial  trainee  model  should  be  designed  using  a 
model  of  the  trainee  performance  based  on  subject  matter  expert  opinion. 
This  early  model  should  depict  the  average  trainee  performance  for  the 
accomplishment  of  key  concepts  and  training  objectives  as  a  function  of 
time-in-training.  When  a  sufficient  number  of  trainees  had  completed  their 

training,  the  model  would  automatically  modify  itself  to  reflect  their 
performance. 

Provisions  should  be  made  in  the  instructor  model  design  to  permit 
combining  the  outputs  of  the  diagnostic  and  trainee  models.  This  enables 
the  diagnosis  for  a  specific  trainee  to  be  based  on  the  average  which  is 
represented  by  the  updated  trainee  model.  Only  when  the  diagnostic  and 
trainee  models  are  adapting  themselves  satisfactorily  should  an  adaptive 
training  curriculum  be  considered. 

In  conclusion  it  is  essential  that  the  training  system  design  team 
plan  for  the  adaptive  process  early  in  the  training  system  design. 

DEVELOPING  THE  INSTRUCTOR  MODEL  DESIGN  REQUIREMENTS 

In  order  to  eventually  specify  what  is  required  of  the  instructor 
model  three  categories  of  requirements  should  be  developed.  These  are: 

Instructional  Requirements  -  those  features  which  relate  to  the 
curricul urn  and  its  control,  performance  measurement  and 
evaluation,  trainee  modeling  and  diagnostics. 

Operating  Requirements  -  those  features  which  relate  to  how  the 
system  will  operate;  modeling,  playback,  freeze,  record  keeping, 
etc . 
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Physical  Requirements  -  those  features  which  relate  to  the 
physical  design  of  the  trainee's  and  human  instructor's  stations. 


A  list  of  each  feature  which  should  be  considered  in  developing  the 
design  requirement  is  shown  in  Table  B-4. 

Curriculum  Structure  -  requirements  should  be  based  on  the  need  for  one 
instruction  path  which  all  trainees  must  follow,  or,  alternatively,  where 
there  are  several  curriculum  paths,  the  requirements  should  be  based  on  the 
ability  and  learning  requirements  of  the  individual  trainee. 

Curriculum  Control  -  requirements  should  be  based  on  the  need  for  manual  or 
automatic  selection  of  the  next  training  module  to  be  presented  to  the 
trainee.  If  the  control  is  manual,  the  trainee  or  human  instructor  will 
make  the  selection.  If  the  control  is  automatic,  the  selection  will  be 
based  on  the  trainee's  past  performance  and  the  training  objective  to  be 
attained.  If  the  trainee  or  instructor  can  disapprove  of  the  automatic 
selection  then  an  override  and  reselection  capability  is  required. 

Curriculum  adaptation  for  a  one  path  curriculum  structure  is  normally 
linear.  In  this  case  selection  of  the  next  training  module  in  the  sequence 
is  made  dependent  on  the  trainee's  performance  for  the  last  module. 

Curriculum  control  can  be  non-linearly  adaptive  if  a  multiple  path, 
criterion-based  branching  curriculum  structure  is  available.  Presentation 
of  the  next  training  module  will  depend  on  the  trainee's  past  performance 
and  higher  order  solution  provided  by  the  trainee  model  and  diagnostic 
model.  Adaptive  curriculum  control  usually  is  designed  to  vary  the  degree 
of  problem  difficulty. 

Curriculum  Organization  -  requirements  should  be  based  on  the  need  for 
modular  lessons  or  blocks  which  relate  to  a  specific  training  objective. 
The  modular  approach  is  preferred  because  of  the  ease  with  which  a  module 
can  be  changed. 

Information  defining  the  training  scenario  should  be  stored  as  an 
integral  part  of  each  lesson  module.  This  information  is  needed  for  the 
simulation  and  event  control  model  (see  Appendix  C). 

PM&E  Requirements  -  should  consider  whether  the  PM&E  subsystem  will  be 
automatic  or  manual  or  both.  If  manual  is  specified,  then  the  presence  of 
a  human  instructor  is  required. 

Behavior  to  be  Measured  -  requirements  should  be  based  on  the  definition  of 
each  variable  and/or  discrete  event  to  be  measured  during  the  conduct  of 
each  lesson  or  during  a  group  of  lessons  which  are  related  to  a  specific 
training  objective. 
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TABLE  B-4.  INSTRUCTOR  MODEL  POTENTIAL  DESIGN  REQUIREMENTS 


INSTRUCTIONAL 

Curriculum  Structure 

Single  path,  multiple  path 
Curriculum  Control 

Manual,  automatic,  instructor/trainee  override 
linear  adaptive,  non  linear  adaptive  (branching), 
forward/backward  jumping,  difficulty  adaptive 

Curriculum  Organization 
Modular  lessons 

Block  of  lessons  for  each  training  objective 
Ease  of  addition  subtraction  and  structural  modification 
Integral  scenario  requirements 
PM&E  Requirements 

Manual,  automatic,  criteria 
Behavior  to  be  Measured 

Common  definition  for  each  training  objective,  defined 
by  each  lesson  plan 

Measurement  Type 

Continuous  measurement  of  selected  variables 
Discrete  events  related  to  time 

Discrete  events  related  to  the  sequence  of  other  events 

Transposition  through  another  system 
(like  speech  recognition) 

(Continued) 
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TABLE  B-4.  (Continued) 


Measurement  Units,  Transforms  and  Algorithms 

Continuous  variables  of  different  dimensions 
Specific  events  (non  dimensional) 

Time  to  complete  tasks 
Comparison  to  criteria  performance 
Translation  to  score(s) 

Trainee  Model 

Rudimentary,  expandable,  transparent,  active 
Diagnostic  Model 

Rudimentary,  expandable,  transparant  active 


OPERATING 

Moding  (on-line) 

Start  up/close  down 
Sign  on/sign  off 

Review  trainees  past  performance/select  training 
Review  previous  training/instruction 
Practice/playback 

Speech  training/speech  test  (voice  interactive  systems  only) 


(Continued ) 
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TABLE  B-4.  (Continued) 


Moding  (off-line) 

Trainee(s)  record  review 
Lesson  content  review 
Trainee  model  research 
Diagnostic  model  research 
Software  review  and  modify 
Records 

Short  term  performance  (on  CRT) 

Last  exercise  performance 
Last  session  performance 
Overall  curriculum  performance 
Prior  training 
Operational  experience 
Class  performance 

PHYSICAL 

Trainee's  Station  Layout 

Operational  equipment  and  controls 
Non-operational  equipment  and  controls 
Instructor's  Station  Layout  (if  required) 
Operational  Equipment  and  controls 
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Measurement  Type  Requirements  -  two  measurement  philosophies  prevail:  1) 
measure  everything  all  the  time  and  select  the  data  for  the  measures 
required  by  a  specific  lesson  performance  evaluation  algorithm  or;  2) 

measure  only  those  variables  and  events  required  by  the  algorithm  when  they 
occur.  The  former  requires  minimal  control  logic  and  should  be  used  when 
only  a  few  common  variables  and  events  are  to  be  measured  throughout  the 
curriculum.  The  latter  requires  substantial  control  logic  and  should  be 

used  when  there  are  many  different  variables  and  events  which  must  be 

measured  throughout  the  curriculum. 

Performance  measurement  in  speech  interactive  training  system 
performance  measurement  will  be  based  on  the  logic  representation  of 

recognized  trainee  utterances.  They  are  time-related  discrete  events  which 
can  contain  variable  measurement  data,  "Turn  Left  Heading  270. "  Left  is  a 
discrete  piece  of  information  and  270  can  be  considered  as  the  value  of  a 
variable  (heading). 

Measurement  Units,  Transforms  and  Algorithms  -  requirements  should  cover 
the  mathematical  requirements  of  performance  measurement.  Commonly  used 
measurement  units  like  knots,  degrees,  etc.  should  be  maintained  throughout 
the  training  system  design.  Common  transforms  like  root  mean  square,  and 
absolute  average,  are  required  to  manage  the  continuous  variable  data. 
Algorithms  are  required  to  transform  the  measurement  data  into  evaluation 
scores. 

Trainee  Model  -  requirements  should  be  included  from  the  design  inception. 
Most  models  will  start  with  a  rudimentary  mechanization  based  on  a 
qualitive  expection  of  the  average  trainee's  performance.  The  model  should 
be  adaptively  expandable  as  it  gathers  information  about  the  performance  of 
actual  trainees  on  the  training  system.  This  adaptive  development  should 
be  transparent  to  the  trainees  until  the  trainee  model  is  judged  as  being 
reliable,  then  it  can  be  used  in  the  diagnosis  of  future  traine*. 

performance. 

Diagnostic  Model  -  requirements  should  be  included  from  the  design 

inception.  Most  models  will  start  with  a  rudimentary  mechani zation  based 

on  tne  development  of  simple  diagnostic  messages  which  relate  to  difficulty 
in  attaining  specific  training  objectives.  The  model  should  be  adaptively 
expandable  based  on  data  about  the  interrelated  difficulties  encountered  in 
attaining  specific  training  objectives.  This  adaptive  development  should 
be  transparent  to  the  trainee  until  the  diagnostic  model  is  judged  as  being 
reliable,  and  then  it  can  be  used  in  the  diagnosis  of  future  trainee 
performance. 

Moding  (on-line)  -  requirements  should  be  based  on  the  overall  training 

strategy  arrived  at  by  the  training/systems  analyst  with  the  operational 
training  community.  Back  up  modes  should  be  included  where  new  technology 
will  incur  the  risk  of  excessive  training  system  down-time. 
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Moding  (off-line)  -  requirements  should  cover  the  needs  to:  1)  access 
individual  trainee  and  class  records;  2)  review  the  curriculum  in  terms  of 
structure,  lesson  content  and  scenario  requirements;  3)  conduct  analysis 
for  the  trainee  and  diagnostic  models;  and  4)  review,  make  changes  to,  and 
edit  the  training  systems  unique  software  (by  qualified  individuals). 

Trainee's  Station  Layout  -  requirements  should  describe  the  equipment 
necessary  to  interface  the  trainee  with  the  training  system.  Facsimiles 
of  actual  operational  equipment  may  be  used  to  provide  operational  realism. 

Instructor's  Station  Layout  -  requirements  should  describe  the  equipment 
necessary  to  interface  the  human  instructor  with  the  training  system.  The 
requirements  may  be  divided  into  manual,  visual,  and  audio  functions. 

IMPLEMENTING  AN  INSTRUCTOR  MODEL  TECHNOLOGY  UPDATE  AND  PROJECTION 

The  purpose  of  this  step  in  the  development  of  the  Instructor  Model 
is:  1)  to  gain  information  for  application  to  a  specific  instructor  model , 
based  on  the  status  of  other  automated  instructor  models;  and  2)  to 
project  this  information  into  the  production  time  frame  for  the  training 
system  under  consideration. 

Descriptive  documentation  is  not  sufficient  to  provide  a 
state-of-the-art  technology  update  for  the  complicated  systems  as  discussed 
herein.  Training/ systems  analysts  also  require  hands-on  experience  with 
the  training  systems  for  an  effective  update  to  occur.  The  following 
technology  features  should  be  observed  in  other  systems: 

1.  Curriculum  structure  and  its  ability  to  be  changed  easily. 

2.  Approach  to  performance  measurement  and  evaluation,  especially  the 
methods  used  to  develop  the  criteria. 

3.  Development  of  higher  order  models  for  diagnosis  and  trainee 
model i ng. 

4.  Other  novel  features  like  the  use  of  personalized  magnetic  cards 
for  sign-on  and  records. 

The  projection  of  instructor  model  technology  into  the  future  should 
consider  what  features  are  available  now  and  what  is  being  designed  for 
utilization  in,  say,  three  years. 

For  example,  the  presentation  of  instructional  material  is  being 
revol utioni zed  by  large  volume  video  discs.  But  the  cost  is  high.  In  a 
three  year  projection,  the  analysts  might  find  that  the  unit  cost  could 
reduce  by  90%,  and  even  though  material  training  cost  will  rise  by  50%,  the 
use  of  video  disc  still  will  be  very  cost  effective. 
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Each  of  the  considerations  for  the  updating  and  projection  of 
technology  for  a  specific  instructor  model  should  be  carefully  documented 
for  future  reference. 

DECIDING  TO  PROCEED  WITH  THE  DESIGN 

The  decision  process  should  be  predicated  on  whether  the  projected 
instructor  model  technology  will  support  a  specific  model's  requirements. 
Each  documented  requirement  should  be  examined  in  this  regard. 
Uncertainties  about  technological  sufficiency  should  be  related  to  their 
overall  impact  on  the  training  system  and  discussed  with  the  operational 
training  community  to  determine  alternative  solutions.  An  example  is  the 
uncertainty  of  meeting  a  training  requirement  for  a  system  with  multiple 
trainee  stations  using  one  instructor  model.  The  outcome  could  be  a 
decision  to  design  the  system  on  a  modular  basis  using  a  distributed 
processing  concept,  starting  off  with  two  trainee  stations  and  adding  more 
when  the  system  bugs  are  resolved. 

Time  is  on  the  side  of  the  training/system  analysts  when  confronted 
with  a  decision  on  a  technological  uncertainty.  An  application  of  this 
experience  would  be  inclusion  of  the  trainee  and  diagnostic  models  in  a 
passive  role  as  a  requirement  pending  further  research  on  this  facet  of 
instructional  technology. 

The  resulting  final  design  requirements  should  be  reviewed  by  the 
operational  training  community  and  should  be  documented  in  detail  for 
future  reference.  (See  "Develop  the  Instructor  Model  Requirement 
Specification  ,  »• ) 

DEVELOPING  THE  INSTRUCTOR  MODEL  OPERATING  AND  HUMAN  FACTOR  DESIGN 

Once  the  decision  to  proceed  with  the  Instructor  Model  design  has 
been  made,  the  functional  operating  and  human  factors  design  should  be 
described  in  detail.  The  resultant  document  will  become  the  preliminary 
functional  description. 

It  is  essential  that  this  description  be  of  sufficient  breadth  and 
depth  so  that  it  meets  the  requirements  of  training  managers  fulfilling  the 
detailed  needs  of  the  software  programmers. 

Seven  significant  subjects  should  be  covered  in  the  description. 
These  are: 

1.  The  curriculum  structure  and  details  of  each  lesson  therein. 

2.  The  performance  measurements  and  evaluations  which  will  be  made 
during  the  conduct  of  each  lesson. 

3.  The  scenario  content  for  each  lesson  (see  the  Simulation  and  Event 
Control  Model  Design  Guide  in  Appendix  C). 
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4.  The  manner  in  which  information,  instruction,  proficiency  testing, 
and  feedback  will  be  presented  to  the  trainee  during  all  on-line 
operating  modes. 

5.  The  file  structure  and  content  format  of  all  trainee,  class  and 
instructor  records. 

6.  The  interface  requirements  with  other  training  system  models  and 
elements. 

7.  The  operation  of  equipment  and  controls  at  the  trainee's  and 
instructor's  stations. 


It  is  recommend  that  these  seven  subjects  be  addressed  in  the  context 
of  working  through  typical  training  scenarios  from  the  time  the  trainee 
(and  instructor)  sign-on  to  the  system  to  sign-off. 

The  functional  design  is  an  exacting  process  that  can  be  expected  to 
reveal  new  instructional  and  human  factors  problems.  No  problem  should  be 
exempted  due  to  difficulty.  Each  must  be  resolved  in  a  timely  manner.  The 
continuous  use  of  operational  training  experts  normally  is  essential  for  a 
functional  development  envisaged  by  this  guideline. 

DEVELOPING  THE  INSTRUCTOR  MODEL  REQUIREMENT  SPECIFICATION 

This  specification  is  the  culmination  of  the  preliminary  design 
process  and  should  be  predicated  on  the  agreed  to  design  requirements  for 
the  instructor  model  and  the  preliminary  functional  description. 

An  organization  for  the  specification  is  shown  in  Figure  B-4.  The 
preliminary  functional  description  should  be  added  as  an  attachment 
thereto.  Furthermore,  the  specification  should  not  be  written  for  a 
general  instructor  model;  it  should  specifically  address  every  known  aspect 
of  the  model  under  consideration. 

Generic  treatment  of  specifications  often  is  resorted  to  when  the 
writers  do  not  fully  understand  the  subject  matter  and/or  do  not  recognize 
when  details  are  required.  This  practice  can  cause  problems  when  the 
specification  becomes  part  of  a  contract.  For  details  which  are 
unavailable  at  the  time  the  specification  is  written,  it  is  prudent  to  use 
the  notation  "TBD"  (to  be  determined)  and  then  to  make  the  "determinations" 
part  of  the  contract. 
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DESIGN  GUIDELINES 


GENERAL  APPROACH 

Figure  B-l  presents  a  design  procedure  intended  to  provide  guidance  to 
training/ systems  analysts  and  systems  engineers  (the  training  system  design 
team)  on  the  implementation  of  an  Automated  Instructor  Model  to  support 
training. 

DEFINE  THE  DEGREE  OF  AUTOMATION  OF  THE  MODEL 

Decide  on  the  extent  of  instructor  automation  which  is  suited  to  the 
training  tasks  and  training  system  being  considered.  Points  which  should 
be  considered  are: 

1.  Instructor  functions  suited  to  automation. 

2.  Instructor  model  development  cost. 

3.  Instructor  manpower  availability  and  cost. 

4.  Relative  training  effectiveness  of  the  automation. 

5.  User  acceptance  and  utilization. 


Define  the  degree  of  instructor  automation  early  in  the  design 
process.  This  is  important  because  of  the  impact  on  the  other  models  and 
subsystems  of  the  training  system. 

DEVELOP  THE  SYSTEM  DESIGN  STRATEGY 

Develop  a  system  design  for  the  instructor  model  to  accomplish  a 
specified  course  of  training  by  working  through  each  training  requirement 
to  arrive  at  a  top  level  system  solution. 

Develop  a  series  of  block  diagrams  which  cover  the  major  elements  of 
the  instructor  model,  tabulating  the  information  which  must  flow  internally 
between  the  blocks  and  interfacing  systems. 

Where  interfacing  equipment  is  involved,  ensure  that  the  required 
information  flow  to  and  from  the  instructor  model  is  available  and  in  a 
manageable  format. 

Define  the  modes  required  to  fulfill  each  training  requirement  and  the 
manual,  visual,  audio  interfaces  between  the  instructor  model ,  the  trainee, 
and  the  human  instructor  (when  required). 
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DEVELOP  THE  CURRICULUM 

Use  the  best  possible  subject  matter  and  instructional  technology 
expertise  to  help  in  the  development  of  the  curriculum. 

Follow  the  ISD  process  to  define  curriculum  organization,  course 
syllabi  and  behavioral  objectives. 

Partition  the  curriculum  into  manageable  modules  to  facilitate 
courseware  development  and  any  subsequent  changes. 

Keep  up  with  changes  in  fleet  training  which  could  affect  the 
curriculum  content  prior  to  the  training  system  entering  into  service. 

DEFINE  THE  PERFORMANCE  MEASUREMENT  AND  EVALUATION  CRITERIA 

Define  the  performance  measurement  and  evaluation  criteria  which  will 
be  used  to  measure  the  performance  of  the  trainee  throughout  the  conduct  of 
his/her  training  on  the  training  system. 

Define  the  parameters  to  be  measured  in  each  lesson  module  including 
continuous  variables,  discrete  events  and  their  sequences. 

Using  subject  matter  experts,  quantify  each  parameter  for  varying 
levels  of  trainee  performance  expected  during  each  lesson  or  sequence  of 
events. 

Decide  on  a  measurement  philosophy.  You  can  measure  everything  and 
extract  the  selected  parameters  for  a  particular  lesson  or  sequence;  or  you 
can  measure  only  the  parameters  which  are  specifically  needed  for  a 
particular  lesson,  event,  or  sequence. 

Using  subject  matter  experts,  develop  a  series  of  performance 
evaluation  algoritihms  concurrently  with  obtaining  the  performance 
measurement  criteria  for  each  lesson  or  sequence  of  events. 

DEVELOP  THE  MODEL  OF  THE  TRAINEE  AND  DIAGNOSTIC  SCHEME 

Develop  two  adaptive  models,  one  which  defines  the  performance  of  the 
average  trainee  as  a  function  of  time-in-training;  the  second  which  will 
provide  simple  diagnosis  of  performance  achievement  of  training  objectives. 
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DESIGN  GUIDELINES 


Design  these  models  to  accumulate  data  based  on  actual  trainee 
performance.  Initially,  neither  model  should  have  any  effect  on  the 
conduct  of  training  or  the  assessment  of  the  trainee .  The  models  should  he 
designed  to  interact  appropriately,  and  to  optimize  their  functions  as 
trainee  data  become  available. 

Make  provision  in  the  design  for  recording  the  adaptation  process  so 
that  it  can  be  reviewed  in  the  off-line  mode. 

DEVELOP  THE  INSTRUCTOR  MODEL  DESIGN  REQUIREMENTS 

Develop  a  set  of  specific  system  design  requirements  based  on  the 
potential  requirements  listed  in  Table  B-4. 

Document  each  requirement  using  a  format  such  that  it  may  be  evaluated  in 
terms  of  available  technology.  (See  also  "Implement  an  Instructor  Model 
Technology  Update  and  Projection.") 

UPDATE  AND  PROJECT  INSTRUCTOR  MODEL  TECHNOLOGY 

Relate  each  instructor  model  design  requirement  to  the  technology 
available  to  fulfill  it.  Take  into  consideration  the  training  system 
production  time  frame. 

Witness  the  operation  of  training  systems  which  use  automated 
instructor  model  technology  to  gain  first  hand  experience  of  their 
performance.  Investigate  other  instructor  models  which  are  in  the  design 
and  development  stage.  Integrate  the  information  obtained  by  these  two 
techniques  into  a  technology  projection. 

Evaluate  other  advancing  technologies  which  have  the  potential  of 
fulfilling  the  instructor  model  design  requirements.  Look  into  its  success 
in  other  applications  and  weight  your  findings  accordingly  and  draw 
conclusions  as  to  its  viability  for  this  specific  training  system. 

DECIDE  TO  PROCEED  WITH  THE  DESIGN 

Make  a  decision  to  proceed  (or  not)  based  on  the  outcome  of  the 
analysis  of  matching  the  requirements  with  the  available  technology. 

Where  the  answer  is  not  clear,  the  operational  training  community  can 
help  to  evaluate  alternative  training  strategies  that  will  fit  within  the 
technology  expectation  of  the  production  time  frame. 
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Reiterate  the  design  requirements  until  there  is  a  feasible  match 
between  the  operational  training  needs  and  what  the  technology  will 
support. 

Document  for  future  reference,  the  final  instructor  model  design 
requi rements. 

DEVELOP  THE  INSTRUCTOR  MODEL  OPERATING  AND  HUMAN  FACTORS  DESIGN 

Develop  a  preliminary  functional  design  and  document  it  in  sufficient 
detail  so  that  diversely  interested  people,  from  training  managers  to 
working-level  programmers,  will  understand  how  the  instructor  model  is 
expected  to  work. 

Within  the  system  operating  context,  cover  the  seven  significant 
subjects  discussed  in  the  previous  section  under  the  heading  "Developing 
the  Instructor  Model  Operating  and  Human  Factor  Design." 

Try  to  resolve  problems  as  they  arise  using  the  assistance  of 
operational  training  experts  where  necessary.  Do  not  hesistate  to  stop  the 
design  process  if  proposed  solutions  will  cause  the  system  design 
requirements  to  be  eroded. 

DEVELOP  THE  DESIGN  REQUIREMENTS  SPECIFICATION 

Develop  the  instructor  model  design  requirements  specification  using 
the  organization  shown  in  Figure  B-4.  The  preliminary  functional  design 
document  should  be  attached  thereto. 

Compose  each  specification  item  in  detail  using  the  abbreviation  "TBD" 
(to  be  determined)  when  specific  data  is  unavailable. 
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SIMULATION  AND  EVENT  CONTROL  MODEL  DESIGN  GUIDE 


INTRODUCTION 

The  design  guidelines  for  the  simulation  and  event  control  model  were 
developed  under  the  assumption  that  the  model  will  operate  with  an 
Automated  Instructor  Model  (see  Appendix  B).  However,  this  may  not  always 
be  the  case.  Where  departure  from  the  guidelines  herein  are  necessary,  the 
document  is  appropriately  annotated. 

The  design  guide  assumes  that  the  user  has  a  working  knowledge  of 
simulation  in  training  systems  and  also  has  detailed  knowledge  of  the 
training  task  to  which  simulation  and  event  control  modeling  is  to  be 
appl ied. 

The  design  guidelines  for  each  section  are  preceded  by  related 
discussion  and  amplifying  comments. 

These  design  guidelines  are  intended  to  encompass  the  following  design 
procedure  as  depicted  in  Figure  C-l. 

1.  Define  the  type  of  information  presentations  which  are  required  to 
meet  the  learning  and  instructional  requirements  set  out  in  the 
curricul urn. 

2.  Analyze  the  information  presentation  requirements  for  a  potential 
system  solution. 

3.  Develop  a  simulation  and  event  control  system  design  to  fulfill 
the  information  presentation  requirements. 

4.  Develop  the  simulation  and  event  control  model  functional  design 
requirements  based  on  the  information  presentation  analysis. 

5.  Assess  the  present  simulation  and  event  control  model  technology 
state-of-the-art  and  project  the  technology  into  the  production 
timeframe  of  the  training  system. 

6.  Make  the  technical  decision  that  the  simulation  and  event  control 
model  requirements  can  be  met  within  the  required  timeframe. 

7.  Develop  the  operating  and  human  factors  design  of  the  simulation 
and  event  control  model,  including  its  interface  with  other 
elements  and  models  which  make  up  the  training  system. 

8.  Develop  the  requirement  specification  of  the  simulation  and  event 
control  model  so  that  it  becomes  an  effective  document  for  the 
designers,  developers,  and  users  of  the  training  system. 
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DEFINING  THE  INFORMATION  PRESENTATION 

In  most  training  system  endeavors,  the  operational  training  community 
can  be  expected  to  attempt  to  dictate  the  information  to  be  presented  in 
the  simulation,  and  where  possible,  the  fidelity  required  of  it.  However, 
in  the  end  the  instructional  content  of  the  curriculum  and  not  subjective 
opinion,  should  dictate  the  information  presentation.  However,  The 
traini ng/ systems  analyst  should  heed  the  operational  communities  ideas  but 
be  prepared  to  market  alternative  approaches.  Using  a  technically 
sophisticated  example  to  illustrate  the  point,  the  operational  people  may 
think  that  they  require  a  wrap-around  point-source  visual  scene  projection 
using  a  high  resolution  model  board.  However,  the  traini rg/system  analysts 
may  decide  that  a  wide  angle  back-projected  TV  screen  using  computer 
graphics  will  fulfill  the  curriculum  requirements.  A  viable  comment  would 
be  that  the  model  board  approach  is  an  expensive  and  inflexible  approach 
compared  to  computer  graphics  (the  advantages  of  CGI  are  an  ongoing 
research  issue).  A  further  but  simpler  example  is  that  the  operational 
community  may  not  have  considered  using  speech  generation  to  provide  a 
verbal  dialog  in  conjunction  with  a  visual  scene  demonstration.  In  the 
view  of  the  trai ni ng/systems  analysts  this  would  eliminate  the  need  for  the 
trainee  to  read  the  instructions  on  a  CRT  terminal  concurrently  with 
viewing  the  demonstration.  Note  that  in  both  examples,  the  presentation  of 
information  to  the  trainee  would  be  arrived  at  by  an  analysis  of  training 
requirements  as  opposed  to  subjective  opinion. 

Therefore,  the  presentation  of  information  really  should  not  be 
decided  upon  until  the  training  curriculum  has  been  developed,  at  least  to 
the  level  of  key  concepts  and  training  objectives. 

Once  there  is  sufficient  curriculum  information  available,  the 
presentation  of  information  can  be  arrived  at  by  resolving  the  contents 
into  visual  and  audible  components.  By  definition  the  audible  components 
are  managed  by  the  speech  system  (see  Appendix  A  for  Speech  System  Design 
Guidelines).  The  visual  components  should  be  further  broken  down  into 
visual  scene  and  information  display  components.  The  latter  may  involve 
CRT  displays,  numerical  readouts  and  actual  operational  equipment. 
Remember  that  the  trainee  should  not  be  expected  to  process  too  much 
information  from  diverse  locations.  In  some  circumstances  it  may  be 
necessary  to  centralize  or  even  condense  the  sources  of  information  and 
consider  eliminating  everything  but  the  visual  scene  during  high  visual 
learning  activities. 

Note  that  unless  the  presentation  of  information  calls  for  a  simple 
approach  (like  a  CRT  screen),  the  final  solution  normally  demands  a  series 
of  good  technical  arguments  and  a  negotiated  settlement. 

ANALYZING  THE  INFORMATION  REQUIREMENTS 

The  curriculum  development  is  expected  to  provide  details  of  the 
scenario  requirements  for  a  block  of  lessons  or  each  lesson  module.  The 
scenario  should  contain  the  visual  scene  and/or  information  details  to  be 


139 


NAVTRAEQUIPCEN  80-C-0057-1 


presented  to  the  trainee.  For  example,  if  the  curriculum  calls  for  the 
scene  to  include  an  aircraft  carrier,  the  information  requirements  should 
include  the  carrier  type,  speed  through  the  water,  aspects  to  the  viewer, 
sea  state,  dynamics  in  six  degrees  of  freedom,  day  or  night,  general 
visibility,  color  characteristics,  etc. 

By  describing  in  detail  each  scene  to  be  displayed,  the  overall 
information  presentation  can  be  defined.  It  is  convenient  to  organize  the 
requirements  into  the  general  categories  of  foreground  and  background 
information.  The  foreground  information  is  primary  information  which  will 
probably  require  realistic  representat i on  and  realistic  dynamic 
performance.  The  background  information  is  all  the  other  information  which 
must  be  displayed.  This  would  include  other  physical  objects  in  the  (such 
as  support  ships,  land  mass,  etc.)  alphanumeric  data  (such  as 
instructions,  speech  recognition  feedback)  and  environmental  effects  (such 
as  reduced  visibility,  twilight,  gunfire,  etc.). 

In  the  LSOTS,  the  perspective  of  the  approaching  aircraft  to  the 
landing  signal  officer  is  a  most  important  factor.  In  the  simulation  of 
the  landing  aircraft  the  correct  perspective  is  achieved  by  information 
resolution  as  seen  by  the  LSO.  Therefore,  the  training/systems  analysts 
must  decide  on  the  visual  resolution  required  to  produce  an  acceptable 

presentation. 

DEVELOPING  THE  SYSTEM  DESIGN  STRATEGY 

The  development  is  intended  to  delineate  what  simulation  features  are 
necessary  to  fulfill  the  information  presentation  requirements. 

The  strategy  is  developed  by  analyzing  each  visual  scene  and 
information  display  requirement  to  arrive  at  a  top  level  system  solution. 
A  good  way  to  express  the  system  solution  is  in  a  block  diagram.  An 

example  is  shown  in  Figure  C-2.  Each  interconnecting  line  between  blocks, 
etc.  should  be  identified  with  the  information  to  flow  along  it.  (See 

Table  B-2,  Appendix  B  for  format  .  " 

To  assist  in  the  development  strategy,  each  information  presentation 
requirement  should  be  analyzed  from  the  aspect  of: 

1.  The  general  scenes  to  be  presented,  including  environmental 

effects. 

2.  The  alphanumeric  and  graphical  data  to  be  presented. 

3.  The  interactions  between  the  vehicle  and  driver. 

4.  The  interactions  between  the  vehicle/dri ver  and  other  objects  in 
the  scene. 

5.  The  impact  which  other  models  and  systems  will  have  on  the 
presentation  of  information. 
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Figure  C-2.  Simulation  Event  Control  Model  Interactions 
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DEVELOPING  THE  MODEL  REQUIREMENTS 

The  requirements  for  the  simulation  and  event  control  must  be 
specified  for  each  training  system  application.  Table  C-l  lists  the 
potential  requirements  which  should  be  considered.  For  ease  of  use  the 
table  has  been  subdivided  as  follows: 


Visual  Scene  Characteri  st i cs  -  the  foreground,  background,  and 
envi ronmental  information  presentations. 


Information  Display 
graphic  data. 


Characteristics 


the  display  of  alphanumeric  or 


Scenario  Generation  and  Housekeeping  -  how  the  information  presented 
to  the  trainee  (and/or  the  human  instructor)  is  compiled  and 
control  1  ed. 


Visual  Scene  Characteristics 

Type  of  Scenes  and  Media  -  Prior  to  the  availability  of  computer  graphics 
for  visual  simulation,  scaled  models  or  films  were  the  only  techniques 
available.  Computer  generated  scenes  are  not  "real"  and  their  training 
value  is  the  matter  of  on  going  research  by  NAVTRAEQUIPCEN  using  the  Visual 
Technology  Research  Simulator  (VTRS).  However,  the  availability  of  high 
resolution,  large  multiple-color  displays  suggests  that  computer  graphics 
will  prevail  for  most  future  training  system  applications. 

Do  not  overlook  that  higher  resolution  displays  are  more  demanding  on 
software  development  and  program  storage  and  this  must  be  traded  off 
against  the  resolution  requirements. 

Type  of  Display  -  the  use  of  a  partially  spherical  dome  is  a  well 
established  technique  for  presenting  a  visual  scene  using  a  point  source 
projection.  The  big  advantage  is  that  the  scene  remains  "real"  for  a  large 
area  around  the  point  source  which  will  accommodate  several  people  involved 
with  training.  However,  this  technique  has  the  disadvantage  of  high 
initial  costs. 

Back  projection  of  wide  angle  screens  is  an  improving  technology.  However, 
large  display  angles  require  the  use  of  edge-on-edge  multiple  displays  set 
on  a  curve,  the  radius  or  focal  point  of  which  becomes  critical  for  the 
user.  Where  high  resolution  perspective  viewing  is  required,  as  used  in 
the  LSO  reverse  display  training  system,  the  trainer’s  use  may  be  limited 
to  one  person  (Hooks  and  McCauley,  1980). 

In  contrast,  the  training/ systems  analysts  should  consider  whether  the 
visual  scene  can  be  provided  with  several  limited  viewing  areas  as 
exemplified  by  the  current  use  of  four  TV  screens  in  the  windows  of  several 
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TABLE  C-I.  SIMULATION  AND  EVENT  CONTROL  MODEL  POTENTIAL  REQUIREMENTS 


VISUAL  SCENE  CHARACTERISTICS 
Type  of  Scenes  and  Media 

Scaled  Model,  computer  generated,  in  combination, 
color,  mono 

Type  of  Display 

Point  source  in  a  dome,  back  projection 
single/multiple  projections,  contoured/flat  surfaces 

Foreground  Objects 

Size,  type,  aspect,  dynamics,  resolution,  lights 
Background  Objectives 

Size,  type,  aspect,  dynamics,  resolution,  lights 
Environmental 

Night,  day,  general  visibility,  lighting, 

sea  state,  cloud  ceiling,  special  effects  (like  gunfire) 


INFORMATION  DISPLAY  CHARACTERISTICS 
Data  to  be  Displayed 

Narrative,  tables,  graphics,  in  combination 
Type  of  Display 

Superimposed  on  visual  screen,  CRT  terminal, 
dedicated  readout,  printout,  or  in  combination 

Special  Features 

Light-pen  touch  selection,  information  display  priorities 


(Continued) 
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TABLE  C-I 
(Continued) 

SCENARIO  GENERATION  AND  HOUSEKEEPING 

Scenario  Generation 

Basic  scenario 
Scenario  detailing 

Vehicle  and  Driver  Model  (when  required) 

Scenario  Monitoring 

Real-time  speech  interaction  monitoring  (when  required) 
Scenario  control 
Visual  scene  status 
Queue  control 

Vehicle/dri ver  model  status 
Scenario  status  monitor 
Recording  and  playback 
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heavy  transport  flight  simulators  {United  Airlines  727  simulators  at 
Denver,  Colorado).  The  window  support  structure  provides  a  natural 
division  between  the  screens  and  the  manipulative  power  of  computer 
graphics  accommodates  the  peripheral  view.  For  alphanumeric  and  graphics 
display  the  technology  is  so  well  advanced  that  no  further  comment  is 
required  (see  any  commercial  computer  graphics  terminal). 

Foreground  Objects  -  for  convenience  these  are  described  as  the  objects 
which  are  central  to  training.  Therefore,  they  should  have  high  fidelity 
in  terms  of  type,  size,  motion  dynamics,  perspective,  resolution  and  other 
character i st i cs  such  as  visible  pertruberances  and  lights.  These 
requirements  should  be  defined  accordingly. 

Background  Objects  -  for  convenience,  these  are  described  as  objects  which 
support  the  forground  objects.  They  can  have  lower  fidelity  than  the 
foreground  objects. 

Environmental  -  these  are  variable  effects  which  can  be  superimposed  on  any 
scene  and  cover  night  and  day  conditions,  prevailing  visibility,  horizon 
definition,  cloud  ceiling,  cloud  types,  sea  state  and  special  visual 
effects  like  gunfire. 

Information  Display  Characteristics 

Data  to  be  Displayed  -  this  covers  all  aphanumeric  data  and  graphics.  It 
is  recommended  that  the  analysts  obtain  sufficient  information  about  the 
scenario  so  that  the  data  to  be  displayed  can  eventually  be  developed  on  a 
scene-by- scene  basis. 

Type  of  Display  -  there  are  numerous  ways  of  displaying  data  to  the 
trainee.  Dedicated  CRTs  are  conmonly  used  especially  when  data  entry 
keyboards  are  included  at  the  trainee's  and/or  human  instructor's  stations. 

The  arrangement  of  the  trainee's  station  is  considered  to  be  part  of 
the  human  factors  design  for  the  Instructor  Model  (see  Appendix  B).  The 
use  of  several  data  displays  must  be  done  judiciously  because  they  can 
distract  the  trainee's  atttention  from  the  primary  viewing  scene. 

It  is  sometimes  practical  to  superimpose  limited  amounts  of  data  on  the 
primary  viewing  screen.  Such  insertions  should  be  unobtrusive  and  should 
be  limited  to  such  items  as  performance  scores  for  the  last  training 
exercise  or  speech  recognition  feedback  messages. 

Scenario  Generation  and  Housekeeping 

Basic  Scenario  Generation  -  three  basic  requirements  should  be  developed  as 
f ol 1 ows : 


1.  Generation  of  the  scenario  for  each  training  task  to  be  taught  in 
acccordance  with  the  definitions  provided  by  the  curriculum 
controller. 
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2.  Accommodation  of  environmental  variables  to  the  basic  scenario  as 
introduced  by  the  trainee  or  human  instructor  to  make  the  task 
easier  or  more  difficult. 

3.  Continuous  detailing  of  the  task  to  ensure  that  no  event  which  is 
planned  to  take  place  within  the  scenario  conflicts  with  any 
other. 

Vehicle  and  Driver  Model  -  is  a  subsystem  which  provides  the  required  range 
of  vehicle  and  driver  types.  The  type  of  vehicle  and/or  driver  may  be 
selected  randomly  or  under  control  of  the  curriculum.  It  is  customary  to 
define  the  vehicle  dynamics  and  the  driver  as  two  separate  models. 

Scenario  Monitoring  -  three  basic  monitoring  functions  should  he  developed 
as  follows: 

1.  Monitor  the  relationship  among  the  scenario,  the  display  contents, 
the  vehi  cl  e/dri ver  model  status,  and  inputs  from  the  speech 
recognition  system  (when  used), 

2.  Provide  event  queue  control  so  that  no  planned  scenario  event 
conflicts  with  any  unplanned  event  provoked  by  the  trainee  or  human 
instructor. 

3.  Control  the  recording  and  playback  of  all  information,  transmitted 
to  and  received  from  the  trainee  and  human  instructor  during  the 
course  of  a  specific  training  session. 


UPDATING  AND  PROJECTING  SIMULATION  AND  EVENT  CONTROL  MODLL  TECHNOLOGY 

The  purpose  of  this  step  in  the  development  of  the  simulation  and 
event  control  model  is:  1)  to  gain  information  for  application  to  a 
specific  simulation  and  event  control  model  based  on  the  status  of  similar 
models;  and  2)  to  project  this  information  into  the  production  time  frame 
for  the  training  system  under  consideration. 

Descriptive  documentation  is  not  sufficient  to  provide  the 
state-of-the-art  technology  update  for  the  complicated  systems  discussed 
herein.  Traini ng/ systems  analysts  also  require  direct  experience  with 
other  training  systems  for  an  effective  update  to  occur.  The  following 
technology  features  should  be  observed  in  other  systems: 

1.  The  type  of  visual  scenes  and  alphanumeric  (or  other)  data  which  is 
being  used  and  the  rationale  for  their  selection. 

2.  The  use  of  common  displays  to  present  different  classes  of 
information. 
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3.  Physical  1  imitations  of  the  displays  -  such  as  acceptable  for  night 
but  marginal  for  day. 

4.  Limitations  of  the  displays  -  like  insufficient  resolution  to 
define  the  shape  of  the  vehicle  at  extended  range. 

5.  In-service  maintenance  difficulties  -  like  bias  shifts  of  adjacent 
display  elements. 


The  projection  of  simulation  and  event  control  model  technology  should 
consider  three  further  points  as  follows: 

1.  The  features  which  can  be  provided  now  but  are  not  cost  effective 
at  the  present  time.  The  question  is  whether  advancing  technology 
will  bring  the  cost  down. 

2.  The  features  which  are  available  now  but  are  mediocre  in 
performance.  The  question  is  whether  advancing  technology  will 
enable  the  performance  to  be  satisfactory. 

3.  Will  the  basic  technologies  being  developed  to  display  information, 
such  as  fiber  optics,  high  resolution,  TV,  etc.,  revolutionize  and 
surpass  the  current  state-of-the-art? 


Information  display  technology  continues  to  advance  so  rapidly  that  it 
may  be  prudent  to  periodically  submit  a  technical  requirement  paper  to 

industry  (world  wide)  to  determine  what  is  likely  to  be  available  for 
future  training  systems. 

Each  of  the  considerations  for  the  updating  and  projection  of 
technology  for  a  specific  simulation  and  event  control  model  should  be 

carefully  documented  for  future  reference. 

DECIDING  TO  PROCEED  WITH  THE  DESIGN 

The  decision  process  should  be  predicated  on  whether  the  projected 
technology  for  simulation  and  event  control  models  will  support  the 

requirements  for  a  specific  application.  Each  documented  requirement 
should  be  examined  in  this  regard.  Uncertainties  about  the  technology 
suffiency  should  be  related  to  their  overall  impact  on  the  training  system 
and  discussed  with  the  operational  training  community  to  determine 

alternative  solutions. 

The  analyst's  job  becomes  particularly  difficult  when  confronted  with  a 
potential  high  technology  solution  which  will  provide  effective  training 
but  which  is  difficult  to  "sell"  to  a  conservative  operational  community. 
In  these  circumstances  it  is  sensible  to  project  both  the  new  and  old 
technologies  in  parallel,  nominating  a  later  decision  date  to  select  one 
technology  depending  on  information  gained  in  the  interim. 
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The  resulting  final  design  requirements  should  be  reviewed  by  the 
operational  training  community  and  should  be  documented  for  future 
reference  (  ee  "Develop  the  Simulation  and  Event  Control  Model 
Specification.") 

In  general ,  time  is  on  the  side  of  the  training/ systems  analysts  when 
confronted  with  a  decision  on  a  technological  uncertainty.  However,  in  the 
case  of  simulation  and  event  control  models,  and  in  particular  the  scene 
display  element,  the  use  of  a  new  technology  is  a  dubious  choice  unless  the 
manufacturer  is  prepared  to  share  the  financial  risk. 

DEVELOPING  THE  SIMULATION  AND  EVENT  CONTROL  MODEL  OPERATING  AND  HUMAN 
FACTORS  DESIGN 

Once  the  decision  to  proceed  with  the  model  design  has  been  made  the 
function  operation  and  human  factors  design  should  be  described  in  detail. 
The  resulting  document  will  become  the  preliminary  functional  description. 

It  is  essential  that  the  description  be  of  sufficient  depth  and 
breadth  to  meet,  on  one  hand,  the  requirements  of  training  managers  (who 
must  plan  how  to  use  the  system)  and  on  the  other  hand,  that  it  fulfills 
the  detailed  needs  of  the  software  programmers.  In  comparison  to  other 
training  system  models  and  subsystems,  the  detail  required  for  the 
simulation  and  event  control  model  is  considered  of  utmost  importance 
because  it  has  the  most  “visibility"  to  the  user. 

Six  significant  subjects  should  be  covered  in  the  description.  These 

are: 

1.  The  content  of  the  visual  scene  for  each  change  in 

foreground/background  objects,  and  environmental  effects.  Where 
practical,  drawings  should  be  used  in  lieu  of  words.  Special 
attention  is  required  to  describe  the  dynamics  of  each  object 
and/or  effect  and  their  interactions. 

2.  The  content  of  the  data  information  presentations  on  a  page-by-page 
basis.  Where  fixed  formats  are  anticipated,  examples  of  content 
should  be  given.  Particular  attention  should  be  paid  to  page 
sequence  after  selection  thru  the  use  of  keyboard  entry,  touch 
controls,  or  light  pens. 

3.  The  vehicle's  dynamic  performance  and  its  relationship  to  the 
driver  model.  The  equations  of  motion  (in  six  degrees  of  freedom) 
for  each  type  of  vehicle  model  to  be  displayed  is  part  of  this 
requirement. 

4.  The  rules  which  will  be  used  to  generate  the  scenario  from  the 
lesson  content  provided  by  the  Instructor  Model.  This  should 
include  the  rules  for  modification  of  the  scenario  by  the  human 
instructor  trainee. 
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5.  T^e  rules  of  order  for  the  scenario  monitor. 

6.  The  rules  for  recording,  freezing  and  redisplaying  (playback)  of 
visual  scene  and  information  data  for  a  specific  training  session. 


It  is  recommended  that  these  six  subjects  be  addressed  in  the  context 
of  working  through  typical  and  atypical  training  scenarios  from  the  time 
the  trainee  (and  instructor)  sign-on  to  the  system  to  the  time  they 
sign-off. 

The  functional  design  is  an  exacting  process  which  can  be  expected  to 
reveal  new  instructional,  display,  and  human  factors  problems.  No  problem 
should  be  exempted  due  to  difficulty.  Each  must  be  resolved  in  a  timely 
manner.  The  use  of  operational  training  experts  normally  is  essential  for 
a  functional  development  envisaged  by  this  guideline. 

DEVELOPING  THE  SIMULATION  AND  EVENT  CONTROL  MODEL  REQUIREMENT  SPECIFICATION 

This  specification  is  the  culmination  of  the  preliminary  design 
process  and  should  be  predicated  on  the  design  requirements  for  the  model 
and  the  preliminary  functional  description. 

An  organization  for  the  specification  is  shown  in  Figure  C-3.  The 
preliminary  functional  description  should  be  added  as  an  attachment 
thereto.  The  specification  should  not  be  written  for  a  general  simulation 
and  event  control  model.  It  should  specifically  address  every  known  aspect 
of  the  model  under  consideration. 

Generic  treatment  of  specifications  is  resorted  to  when  the  writers  do 
not  fully  understand  the  subject  matter  and/or  recognize  when  details  are 
required.  This  practice  can  cause  problems  when  the  specification  becomes 
part  of  a  contract.  For  details  which  are  unavailable  at  the  time  the 
specification  is  written,  it  is  prudent  to  use  the  notation  "TBD"  (to  be 
determined)  and  then  to  make  the  "determinations"  part  of  the  contract. 


simulation  and  event 

CONTROL  MODEL 
RLOUIR£M£HT  SPEC  IF! CAT  I  ON 


Figure  C-3.  Simulation  and  E’.ent  Control  Mode1  Requirement  Specification  Organization 
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DESIGN  GUIDELINES 


GENERAL  APPROACH 

Figure  C-l  presents  a  design  procedure  intended  to  provide  guidance  to 
training/ systems  analyst  and  system  engineers  (the  training  system  design 
team)  on  the  implementation  of  an  automated  simulation  and  event  control 
model  to  support  training. 

ANALYZE  THE  INFORMATION  REQUIREMENTS 

Break  each  visual  scene  into  three  components  (foreground  objects, 
background  objects,  and  effects),  ensuring  there  is  a  good  understanding  of 
their  characteristics  and  interactions. 

Define  the  set  of  common  objects,  unique  objects  and  effects  which 
have  to  be  simulated. 

Define  the  dynamics  and  resolutions  required  for  the  foreground  and 
background  objects. 

DEFINE  THE  INFORMATION  PRESENTATION 

Allow  sufficient  time  for  the  key  training  concepts  and  training 
objects  to  be  developed  as  part  of  the  curriculum  design  before  proceeding 
with  the  information  presentation  definition. 

Resolve  the  curriculum  information  into  visual  scene  and  information 
display.  Allocate  the  information  display  to  CRTs,  dedicated  readouts  and 
operational  equipment. 

Guard  against  the  display  of  too  much  information  to  the  trainee  at 
any  instant. 

Do  not  allow  preconceived  notions  of  how  the  visual  scene  and  data 
information  should  be  presented  to  the  trainee  to  prempt  the  orderly 
analysis  and  progress  towards  the  best  solution. 

DEVELOP  THE  SYSTEM  DESIGN  STRATEGY 

Develop  a  top  level  system  design  that  will  provide  the  simulation  and 
presentation  of  information  to  fulfill  scenario  needs. 
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DESIGN  GUIDELINES 


Develop  a  series  of  block  diagrams  to  describe  the  major  elements  of 
the  simulation  and  event  control  model,  tabulating  the  information  which 
must  flow  internally  between  the  blocks  and  interfacing  systems. 

Where  interfacing  systems,  e.g.,  an  existing  visual  display,  ensure 
that  the  required  information  flow  to  and  from  the  simulation  and  event 
control  model  is  available  and  in  a  manageable  format. 

DEVELOP  THE  MODEL  REQUIREMENTS 


Develop  a  set  of  specific  design  requirements  based  on  the  potential 
requirements  listed  in  Table  C-l. 

Document  each  requirement  using  a  format  so  that  it  may  be  evaluated 
in  terms  of  available  technology. 

UPDATE  AND  PROJECT  SIMULATION  AND  EVENT  CONTROL  MODEL  TECHNOLOGY 

Relate  each  specific  model  requirement  to  the  technology  available  to 
fulfill  it.  Take  into  consideration  the  training  system  production  time 
frame.  Weight  your  findings  accordingly. 

Witness  the  operacion  of  recently  developed  training  systems  to  gain 
first  hand  of  their  simulation  and  event  control  performance.  Firmly 
establish  the  expectations  of  other  simulation  efforts  which  are  in  the 
design  and  development  stage  and  weight  your  findings  accordingly. 

Evaluate  other  advancing  information  display  technologies  which  have 
the  potential  of  fulfilling  the  simulation  and  event  control  model  under 
consideration.  Look  into  their  success  in  other  applications  and  draw 
conclusions  as  to  their  viability  for  the  specific  training  system 
application  being  considered. 

DECIDE  TO  PROCEED  WITH  THE  DESIGN 

Make  a  decision  to  proceed  (or  not)  on  the  outcome  of  the  analysis  of 
matching  the  requirements  with  the  available  technology. 

When  the  answer  is  not  clear,  the  operational  training  community  can 
help  to  evaluate  alternative  training  strategies  that  will  fit  within  the 
technology  expectations  of  the  production  time  frame. 
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DESIGN  GUIDES 


Reiterate  the  design  requirements  until  there  is  a  feasible  match 
between  operational  training  needs  sand  what  the  technology  will  support. 

Do  not  hesitate  to  consider  an  alternative  candidate  for  the  visual 
scene  display  if  the  newer  technology  looks  promising.  Set  a  date  to  make 
a  final  decision  on  which  technology  to  use. 

Document  for  future  reference,  the  simulation  and  event  control  model 
design  requirements. 

DEVELOP  THE  OPERATING  AND  HUMAN  FACTORS  DESIGN 

Develop  a  preliminary  functional  design  and  document  it  in  sufficient 
detail  so  that  diversely  interested  people  from  training  managers  to 
working  level  programmers  will  understand  how  the  Simulation  and  Event 
Control  Model  is  expected  to  work. 

Within  the  system  operating  context,  cover  the  six  significant 
subjects  discussed  in  the  previous  section  on  operating  and  human  factors 
design. 

Try  to  resolve  problems  as  they  arise  using  the  assistance  of 
operational  training  experts  where  necessary.  Do  not  hesitate  to  stop  the 
design  process  if  the  proposed  solutions  will  cause  the  system  design 
requirements  to  be  eroded. 

DEVELOP  THE  DESIGN  REQUIREMENTS  SPECIFICATIONS 

Develop  the  Simulation  and  Event  Control  Model  requirements 
specification  using  the  organization  shown  in  Figure  C-2.  The  preliminary 
functional  design  document  should  be  attached  thereto. 

Compose  each  specification  item  in  detail  using  the  apprevation  "TBD" 
(to  be  determined)  when  specific  data  is  unavailable. 
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NAVTRAEQUIPCEN  80-C-00b/-l 


TABLE  OF  CONTENTS 


INTRODUCTION. 


SYSTEM  INTEGRATION  DESIGN. 


EXAMPLES  OF  TRAINING  SYSTEM  INTEGRATION. 

Speech  Procedure  Systems . 

Task  Procedure  Systems . 

Control  Strategy  Systems . 


CONCEPTS  FOR  THE  TRAINING  SYSTEM  EXECUTIVE 
CONTROL  DESIGN . 


DEFINING  THE  INFORMATION  FLOW  BETWEEN 
SYSTEM  ELEMENTS . 


DEVELOPING  A  FUNCTIONAL  SOFTWARE 
REQUIREMENT  SPECIFICATION . 


DESIGN  GUIDELINES. 


Figure 


Figure 


LIST  OF  ILLUSTRATIONS 


D-l.  Self-Contained,  Voice  Interactive 
Training  System:  Subsystem 
Relationships  and  Executive 
Control . 

D-2.  Alternative  Training  System 

Strategies . 


Figure  D-3.  Primary  System  Categories. 


Figure 


Figure 


Figure 


Figure 


D-4.  The  Importance  oc  the 

Training  System  Executive. 


D-5.  Data  Flow  Between  System 

Elements  for  A  Candidate  LSOTS..., 

D-6.  Typical  Computing  Function 

Description . . 


D-7.  Example  of  Part  of  an 
Event  Sequence  Map.... 


NAVTRAEQUIPCEN  80-C-0057-1 


Table  D- 

Table  0- 


LIST  OF  TABLES 


Page 


.  Points  to  be  Considered  When 
Designing  a  Training  System 


Executive .  166 

.  Training  System  Executive 

Analysis  Tasks . . .  172 


157 


NAVTRAEQUI PCEN  80-C-0057-1 


AUTOMATED  TRAINING  SYSTEM  INTEGRATION 
DESIGN  GUIDE 


INTRODUCTION 

The  design  guidelines  for  automated  training  system  integration  are 
oriented  toward  the  relationship  among  an  automated  speech  system, 
instructor  model,  and  simulation  and  event  control  model.  These  guidelines 
also  address  the  design  considerations  when  only  two  of  the  foregoing 
training  system  elements  are  used,  or  in  some  instances,  when  only  one 
system  element  is  used  with  an  existing  simulator  or  trainer. 

The  design  guide  assumes  that  the  user  has  a  working  knowledge  of 
speech  systems,  instructor  models,  and  simulation  and  event  control  model 
designs  and  the  interrelationships  between  them. 

The  design  guidelines  for  each  section  are  preceded  by  related 
discussion  and  amplifying  comments. 

System  integration  is  a  broad  subject.  For  this  design  guide  the 
following  subjects  are: 

1.  Automated  training  system  integration  design  -  system 
partitioning,  and  i n ter faces  with  existing  simulators  or 
trainers. 

2.  Examples  of  training  system  integration  design  -  existing  and 
proposed. 

3.  Concepts  for  the  design  of  training  system  executive  controls  . 
SYSTEM  INTEGRATION  DESIGN 

The  present  partitioning  of  systems  and  models  used  in  automated 
training  system  design  largely  has  occurred  along  the  same  lines  as  the 
technology  development  (speech,  instructional ,  and  information  display). 

Figure  D-l  shows  the  global  system  concept  of  using  three  "satellite" 
systems  working  under  the  direction  of  a  training  system  executive  control. 
Each  of  the  "satellites"  is  a  self  contained  system  with  its  own  executive 
control . 

Speech  interactive  training  systems  like  PARTS  and  ACE  have  been 
designed  to  provide  the  functions  of  the  three  satellites  (speech, 
instruction,  and  information  display)  without  a  recognizable  training 
system  executive  control  system.  In  these  designs,  multiple  computers  and 
peripherals  have  been  used  to  mechanize  the  system.  This  technique  is  not 
conducive  to  functional  change.  More  importantly,  it  is  difficult  to  use 
this  system  integration  experience  to  design  systems  using  different 
combinations  of  the  satellites. 
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Figure  D-l.  Self-Contained,  Voice  Interactive  Training  System: 
Subsystem  Relationships  and  Executive  Control 
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Different  combinations  of  the  satellite  system  (see  Figure  D-2)  can  be 
considered  as  follows: 

1.  Instructor  Model  with  a  Speech  System  working  with  an  existing 
simulator. 

2.  Simulation  and  Event  Control  Model  with  a  speech  system  working 
with  an  existing  instructor's  console  and  display  system. 

3.  Instructor  model  working  with  an  existing  simulator. 

4.  Simulation  and  event  control  model  working  with  an  existing 
instructor  console  and  display  system. 

5.  Instructor  model  in  a  computer  assisted  instruction  application. 


Examples  of  integration  of  the  various  training  system  elements  are 
given  in  this  document.  The  training  system  design  team  is  cautioned  about 
the  considerations  which  must  be  given  to  the  integration  of  the  instructor 
model,  simulation  event  control  model,  and  speech  system  with  an  existing 
simulator  or  piece  of  training  equipment.  Details  of  the  functional  and 
electronic  characteristics  of  the  proposed  interface  with  the  existing 
system  should  be  carefully  examined  before  the  feasibility  of  system 
integration  can  be  determined. 

EXAMPLES  OF  TRAINING  SYSTEM  INTEGRATION 

The  nature  of  a  training  system  functional  integration  is  dependent  on 
the  tasks  to  be  learned.  There  are  innumerable  variations  that  can  bo 
discussed.  However,  there  are  three  primary  system  categories  which  are 
depicted  in  Figure  D-3  and  summarized  as  follows: 

Speech  Procedures  Driven  Systems  -  in  which  the  verbal  communication 
between  the  automated  system  and  the  trainee  is  the  focal  point  of 

training.  Without  a  primary  need  to  develop  speech  skills,  the  training 
system  would  not  exist.  Existing  examples  of  this  type  of  system  are 

PARTS,  ACE,  and  LSOTS. 

Task  Procedure  Driven  Systems  -  in  which  the  training  system  primarily 

is  used  to  develop  these  skills  and  may  or  may  not  include  speech 

interaction.  An  example  of  this  type  of  system  is  the  F-14A  Instructor 
Support  System  (F-14  ISS). 

Control  Strategy  Driven  System  -  in  which  the  training  system  is  used 
to  develop  trainee  information  selection  strategies  based  on  discrete 
inputs  by  the  trainee.  The  inputs  could  be  by  keyboard  or  voice.  An 
example  of  this  type  of  system  is  a  troubleshooting  system  for  maintenance 
training. 
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Figure  D>2.  Alternative  Training  System  Strategies 
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Figure  D-3.  Primary  System  Categories 
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Speech  Procedures  Systems 

These  v'-tems  are  critically  interactive  because  the  training  system 
control  loop  is  closed  through  the  trainee.  Except  for  the  instruction 
feature,  the  training  system  must  react  in  a  real  time  situation.  For  each 
trainee  speech  utterance,  the  training  system  should  provide  feedback.  The 
allowable  time  for  each  training  exercises  to  take  place  (30  second 
exercises  at  1  1/2  minute  intervals  for  LSOTS),  imposes  a  heavy  burden  on 
the  system  designers  to  integrate  the  system  element  and  models  so  that  the 
commensurate  information  flow  is  assured.  In  some  instances  the  system 
integration  design  will  impact  the  operation  of  the  system  elements  and 
models  which  it  supports.  For  example,  in  the  LSOTS,  the  scenario  sequence 
details  for  each  training  exercise,  could  be  compiled  in  the  instructor 
model.  However,  due  to  anticipated  timing  constraints  of  moving  the  data 
to  the  simulation  event  control  model,  the  training/systems  analyst 
selected  to  store  the  scenario  details  as  part  of  the  latter  model. 

Task  Procedures  Systems 

These  systems  are  interactive  at  the  task  execution  level  which  means 
that  the  trainee  and  the  vehicle/dynamics  constitute  the  control  loop. 
The  instructor  model  and  speech  recognition  and  generation  system  are 
convenient  additions  to  the  training  system  to  eliminate  the  need  for  a 
human  instructor.  The  feedback  provided  to  the  trainee  is  the  response  of 
the  vehicle.  In  this  type  of  system  integration,  the  vehicle  dynamics  and 
the  trainee  reaction  become  the  critical  factors  to  be  considered  for  data 
transfer  design. 

Control  Strategy  Systems 

These  systems  are  interactive  at  the  trainee  level  but  differ  from 
speech  procedure  driven  systems  because  the  trainee/system  interaction  is 
designed  as  an  iterative  process  to  arrive  at  a  satisfactory  conclusion. 
These  systems  can  be  keyboard  or  voice  driven.  After  an  entry  is  made  by 
the  trainee,  there  is  a  permissible  finite  time  for  the  training  system  to 
react.  Each  training  system  reaction  provides  feedback  to  the  trainee. 
The  reaction  may  be  a  discrete  event  like  moving  a  symbol  on  a  display 
screen,  or  a  interrogative  response  which  asks  the  trainee  for  more  data  or 
to  make  a  choice.  The  importance  here  is  that  an  ongoing  interactive 
training  format  requires  the  instructor  model  design  to  have  a  large 
hierarchical  structure.  However,  the  data  flow  requirements  normally  are 
not  taxing. 

CONCEPTS  FOR  THE  TRAINING  SYSTEM  EXECUTIVE  CONTROL  DESIGN 

The  primary  purpose  of  the  training  system  executive  (TSE)  is  to 
maintain  a  functional  and  rational  interrelationship  between  the  various 
elements  of  the  training  system  at  all  times  as  depicted  in  Figure  D-4.  It 
is  the  "executor." 
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Figure  0-4.  The  Importance  of  the  Training  System  Executive 
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For  a  self  contained  training  system  as  depicted  in  Figure  D-l  TSE  is 
envisioned  to  provide  inter  model/subsystem  control  for: 

o  System  operating  modes 
o  Curriculum  control 
o  Active  task  selection 

o  Trainee  and  human  instructor  inputs  and  outputs 
o  Simulated  environmental  condition 
o  Vehi cl  e/dri ver  model  or  equivalent  body  dynamics 
o  Performance  measurement  and  evaluation 
o  Diagnosis  of  the  trainee  performance 
o  Storage  of  performance  data  of  the  active  task 
o  Playback  of  the  active  task 
o  Filing  of  trainee  instructor  and  class  records 
o  Speech  data  collection 
o  Speech  recognition 
o  Speech  generation 


The  design  concept  for  the  TSE  is  that  each  system  element  or  model  of 
a  training  system  has  its  own  internal  executive  control  (see  Figure  D-l 
and  D-2).  It  is  then  the  responsibility  of  the  training  system  executive 
to  manage  the  information  flow  between  the  satellite  control  systems  when 
"flagged"  to  do  so.  This  concept  supports  the  increasing  use  of  multiple 
microprocessors  for  distributed  processing  of  large  computing  tasks. 
However,  when  a  mainframe  computer  and  a  group  of  interfacing  peripheral 
equipment  must  be  used,  the  TSE  is  conceived  as  interacting  directly  with 
the  computer's  resident  operating  program.  Additionally,  the  TSE  is 
conceived  as  providing  the  interface  with  existing  simulators  or  training 
equipment  using  microprocessors  for  information  buffering. 

Points  to  be  considered  in  the  design  of  a  TSE  for  a  specific  training 
system  application  are  shown  in  Table  D-l.  The  design  should  be  predicated 
on  loose  coupling  between  systems.  This  enables  the  processing  of  raw 
information,  the  flow  of  which  is  controlled  by  the  TSE,  to  become  the 
responsibility  of  the  system  element  or  model  receiving  it.  This  concept 
allows  each  system  or  model  to  control  processing  in  its  own  internal  time 
frame,  and  in  the  most  efficient  manner. 

DEFINING  THE  INFORMATION  FLOW  BETWEEN  SYSTEM  ELEMENTS 

The  following  steps  should  be  used  in  defining  the  degree  of 
integration  of  the  training  system  design. 

1.  Define  every  computing  function  to  be  used  in  the  training 
system  (see  Figure  D-5  as  an  example).  Identify  which  functions 
have  critical  timing  requirements,  establish  whether  the  data 
resulting  from  the  computation  remains  internal  to  the  system 
element,  or  is  outputted  to  another  system  element. 
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TABLE  0-1. 

TSE  Tasks 

Scheduling 

Monitoring 

Rules  of  Operation 
Priorities 
System  Security 


POINTS  TO  BE  CONSIDERED  WHEN 
DESIGNING  A  TRAINING  SYSTEM 
EXECUTIVE 


-  Definition,  analysis  of  content, 
integration  with  other  models  and 
systems 

-  Realtime,  near  realtime,  offline, 
hard  copy,  time  sharing  for  multi¬ 
ple  trainee  participation 

-  Exercise  sequencing,  unnatural 
circumstances,  self  test, 
diagnostics 

-  General  flow,  special  cases, 
design  conventions 

-  Message  cueing,  message  bypass 
hierarchies 

-  Sign  on/off,  beating  the  system, 
record  protection 
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Figure  D-5.  Data  Flow  Between  System  Elements  for 
a  Candidate  LSOTS 
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2.  Define  the  internal  computing  functions  to  be  controlled  by  each 
system  or  model  internal  executive  control,  including  its  timing 
priority  and  data  format. 

3.  Define  the  "external  to"  or  system  input/output  data  to  be 
managed  by  the  TSE,  including  its  timing  priority  and  data 
format.  Specify  priority  interrupts  as  necessary. 

4.  Develop  a  document  which  describes  how  each  system  element  is  to 
be  integrated  with  other  systems  and  models.  Include  the  data 
processing  requirements  internal  to  each  element  and  data  flow 
between  elements. 

Each  computing  function  should  be  described  using  a  standard  format  an 
example  of  which  is  shown  in  Figure  D-b.  Where  the  system  size  is  not 
overwhelming  the  computational  flow  can  be  developed  by  using  a  modified 
PERT  chart  approach.  (See  Figure  D-7.) 

Experience  with  PARTS  and  ACE  emphasizes  the  need  to  establish 
realistic  timing  priorities  so  that  the  trainee  operates  in  a  environment 
conducive  to  efficient  training.  It  is  very  necessary  that  the  training 
system  visibly  or  audibly  react  to  a  trainee  input  within  1  second.  It  is 
unreasonable  and  frustrating  if  the  trainee  has  to  wait  10  seconds  because 
’ess  visible  software  programs  are  given  priority.  These  days,  prolonged 
waiting  is  probably  the  outcome  of  inadequate  software  processing  priority 
design,  not  inadequate  computer  capability.  It  is  essential  that  the 
training  systems  analysts  establish  the  overall  training  system  priorities 
early  in  the  design  process.  These  priorities  should  be  based  on 
optimizing  the  trainee's  interaction  with  the  system.  An  organizing  order 
for  timing  priorities  is  as  follows: 

Priority  Interrupts  -  All  other  computing  functions  cease  pending 
completion  of  the  priority  interrupt  computing. 

Interrupts  -  Allows  all  other  computing  functions  to  continue  to  a 
convenient  point  (no  longer  than  10  milliseconds)  and  then  to  cease  pending 
completion  of  the  interrupt  processing. 

Foreground  Computation  -  Involves  a  group  of  programs,  the  execution 
of  which  is  essential  for  the  conduct  of  the  training  task.  For  example, 
performance  measurement,  visual  scene  updating,  etc.  The  execution  of  the 
foreground  computations  has  a  priority  structure  unto  itself  so  that 
essential  computing  is  done  ahead  of  computations  of  lesser  importance. 
Foreground  computation  is  surpassed  by  the  priority  interrupt  and  interrupt 
programs. 

Background  Computation  -  Involves  a  group  of  programs  the  execution  of 
which  is  nonessential  for  the  conduct  of  the  training  task  at  hand.  For 
example,  adaptive  revision  of  the  trainee  model,  filing  the  execution  of 
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PROCEDURES  MONITOR 

FUNCTION: 

This  process  will  determine  the  correctness  of  various 
cockpit  procedures  based  on  a  predetermined  procedural 
template.  Pilot  actions,  changes  in  instrument 

readings,  and  touch  pad  inputs  will  provide  the  input 
from  which  procedural  steps  are  derived.  Inputs  will 
be  described  as  discrete  events  and  will  be  detected  by 
the  central  event  detector. 

This  process  will  be  created  by  the  Instructor  Control 
process  at  the  start  of  an  exercise  and  will  be 
terminated  at  the  completion  of  the  exercise.  It  will 
not  be  involved  with  mission  debriefing  or  replay,  only 
mission  execution. 

Events  will  be  checked  for  legality  within  the  current 
procedural  context  and  for  correct  order  of  occurrence. 

One  or  more  activities  may  be  associated  with  the 
occurrence  of  a  single  event.  Activities  may  be  made 
contingent  upon  the  occurrence  or  non-occurrence  of 
previous  events.  This  mechanism  will  allow  the 

definition  of  procedural  completeness  to  be  dynamically 
modifiable  dependent  on  prior  procedural  steps.  This 
contingency  logic  also  means  that  a  pilot's  procedures 
score  will  be  based  on  a  flexible  template  which  can 
adjust  for  changes  in  aircraft  configuration  or 
environment  rather  than  on  a  fixed  procedural 
definition. 

The  Procedures  Monitor  will  support  a  separate  monitor 
task  for  each  active  task  module.  These  tasks  will  run 
asynchronously,  each  task  receiving  event  occurrence 

IPC  messages.  A  single  body  of  reentrant  code  will  be 
shared  by  all  monitor  tasks  in  performing  monitoring 
logic  but  a  separate  data  structure  representing  the 
procedural  template  will  be  maintained  by  each  monitor 
task  in  an  unshared  data  area. 

PRIORITY: 

6 

TYPE: 

Preemptible 

FREQUENCY 

Less  than  1/sec.  asynchronous 

STORAGE : 

12  K  shared 

9  K  dedicated 

Figure  0-6.  Typical  Computing  Function  Description 
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Figure  D-7.  Example  of  Part  of  an  Event  Sequence  Map 
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the  background  computation  has  a  priority  structure  unto  itself  so  that  the 
more  important  computation  is  done  ahead  of  computations  of  lesser 
importance. 

DEVELOPING  A  FUNCTIONAL  SOFTWARE  REQUIREMENT  SPECIFICATION 

Table  D-2  should  be  used  to  develop  the  software  functional 
requirement  specification.  Note  that  when  the  information  flow  between 
system  elements  is  properly  defined  (see  defining  the  information  flow 
between  system  elements)  the  outcome  will  provide  the  greater  majority  of 
the  information. 

Two  later  additions  to  the  specification  should  be  provided  for:  1) 
the  description  of  the  interface  between  the  TSE  and  the  vendor-supplied 
standard  operating  program  and;  2)  the  description  of  the  interface  between 
the  simulation/event  control  executive  control  program  and  the 
vendor- supplied  visual  scene  display  servicing  program.  Neither  program 
can  be  completely  specified  until  systemware  is  selected.  However,  an 
early  and  well  defined  description  of  the  interface  as  seeen  by  the  TSE  and 
the  simulation/event  control  executive  is  a  prerequisite  to  a  satisfactory 
design. 
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TABLE  0-2. 


Requi rements 


Organization 


Rules  of  Operation 


TRAINING  SYSTEM  EXECUTIVE 
ANALYSIS  TASKS 


-  Operations  With  Other  Subsystems 

-  Operating  Tasks 

-  Sequencing 

-  Security 

-  Partitioning 

-  Scheduling 

-  Tasks  Allocated  to  Software 

-  Tasks  Allocated  to  Hardware 

-  Tasks  Allocated  to  Courseware 

-  Time  Sharing 

-  Interfaces  With  Unique  User  Equipments 

-  General  Flow  of  Information 

-  Special  Information  Flows 

-  Message  Cueing  and  Bypass 

-  Housekeeping/Self  Test 

-  Records 
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DESIGN  GUIDELINES 


DEFINE  THE  SYSTEM  INTEGRATION  REQUIREMENTS  AT  A  GLOBAL  LEVEL 

Partition  the  system  functions  to  minimize  the  information  flow 
between  major  parts  of  the  training  system. 

Ensure  that  when  a  new  training  system  is  to  be  integrated  with  an 
existing  piece  of  training  equipment  the  resultant  interface  design  is 
manageable. 

Examine  any  critical  timing  constraints  imposed  by  the  training 
scenarios  to  ensure  that  data  flow  between  system  elements  is  manageable. 

DEFINE  THE  INFORMATION  FLOW  BETWEEN  SYSTEM  ELEMENTS 

Define  every  computing  function  to  be  conducted  in  the  training 
system. 

Identify  which  system  element  or  model  is  responsible  for  the 
computation. 

Describe  whether  the  computation  has  input/output  data  requirements  - 
if  so,  describe  the  data  format  and  the  timing  priorities  to  be  established 
by  the  TSE. 

Document  the  foregoing  information  as  part  of  the  functional  software 
description. 

DEVELOP  A  FUNCTIONAL  SOFTWARE  REQUIREMENT  SPECIFICATION 

Develop  a  formal  specification  which  describes  the  overal'  software 
requirement  fo1  the  training  system. 

Document  the  computing  functions  to  be  conducted  in  each  system 
element  or  model  and  the  data  interchange  between  them.  Computational, 
input/output  timing,  and  data  storage  (permanent  and  temporary) 
requirements  should  be  established  for  each  computation. 

Identify  common  operating  programs  which  are  required  to  be  resident 
in  the  computer  system  to  support  the  training  system  software  (files, 
software  update,  self  test,  diagnostic,  etc.). 
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APPENDIX  E 

DESIGN  AND  DEVELOPMENT  EVALUATION 
GUIDELINES 
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INTRODUCTION 

The  design  and  development  evaluation  document  is  a  compendium  to  the 
design  guides  presented  in  Appendices  A  through  D.  It  came  into  being 
because  the  authors  strongly  believe  that  design  evaluation  followed  by 
developmental  test,  evaluation  and  revision  are  the  cornerstones  of 
successful  system  development.  A  proposed  OPNAV  Instruction  (Cordell, 
Nutter,  and  Heidt,  1979)  is  recommended  as  the  guiding  document  for  the 
test  and  evaluation  process  applied  to  training  systems. 

The  present  document  assumes  that  the  user  has  a  working  knowledge  of 
interactive  training  system  design  principles,  a  detailed  knowledge  of 
the  specific  training  task,  and  is  familiar  with  the  training  system 
design  team's  endeavors.  The  content  is  divided  into  two  sections, 
addressing  system  design  and  system  development  respectively.  We 
recognize  that  this  distinction  may  not  be  clear  for  all  systems. 

Evaluating  the  design  of  a  large  complex  training  system  at  any  stage 
in  the  design  and  development  process  is  a  difficult  problem  which  must 
be  addressed.  Training  systems  which  are  designed,  developed  and  built 
under  a  judicious,  well  organized  design  evaluation  policy  will  be  more 
successful  in  the  operational  environment. 

Design  evaluation  normally  is  an  ongoing  practice  of  good  designers, 
whatever  their  discipline.  Design  evaluation  should  be  practiced  on  a 
day-to-day  basis  within  the  design  team.  It  should  also  be  expanded  to 
include  scheduled  design  reviews  by  other  people  with  related  expertise, 
at  which  time  the  design  team  justifies  the  design  to  their  peers. 
Critical  matters  can  then  be  carried  over  from  one  review  to  the  next. 

PRINCIPLES  OF  ONGOING  DESIGN  EVALUATION 

Many  projects  suffer  because  there  is  no  design  evaluation  structure, 
insufficient  design  review,  and/or  the  reviews  are  not  conducted  in  a 
diligent  manner.  Consequently,  problems  that  are  not  solved  quickly 
become  neglected  under  the  pressures  of  maintaining  a  design  schedule. 
The  revelation  of  the  problem  invariably  occurs  during  system  test. 
Design  evaluation  is  an  iterative  process  that  has  its  genesis  within 
the  design  team.  It  includes  higher  level  design  reviews  conducted  by 
peers  to  detect  errors  in  the  lower  order  processes.  However,  the 
project  manager  must  assure  that  the  design  evaluation  requirement  does 
not  consume  project  resources  to  the  detriment  of  the  design  process. 

As  the  design  proceeds,  the  evaluation  process  leads  to  higher  level 
reviews  which  normally  are  conducted  by  the  users,  sponsors  and  project 
management.  These  higher  level  reviews  should  bo  based  on  a 
hierarchical  structured  review  process  that  has  been  established  in 
advance.  It  is  the  responsibility  of  the  reviewers  to  assure  that  the 
lower  processes  have  been  adequately  designed,  and  to  help  resolve  any 
outstanding  problems. 
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DESIGN  EVALUATION  GUIDELINES 


1.  A  design  evaluation  checklist  should  be  established  at  the 
beginning  of  the  project  and  used  throughout  the  design  phase.  An 
example  of  organization  for  the  evaluation  is  shown  in  Figure  E-l. 
It  requires  a  large  initial  effort  by  the  project  manager  to 
document  in  detail  the  design  process  from  the  initial  design 
phase  through  the  end  of  system  test.  Obviously,  the  organization 
is  subject  to  change  as  the  system  design  and  development  unfolds. 

2.  Each  member  of  the  design  team  should  bear  the  responsibility  of 
reviewing  his/her  own  work  and  the  work  of  other  collegues  on  a 
daily  basis. 

3.  Each  week  the  design  team  should  evaluate  the  current  design  tasks 
in  a  group  meeting.  The  person  responsible  for  the  task  must 
justify  his/her  design  decisions  to  all  concerned. 

4.  A  peer  design  review  team  should  evaluate  groups  of  completed 
tasks  and  reiterate  perceived  problems  through  the  design  team. 
The  peer  design  review  team  should  sign  off  on  each  design  task 
documenting  their  remarks  where  necessary  for  the  information  of 
the  project  manager. 

5.  The  project  manager  should  organize  and  conduct  higher  level 
design  reviews  with  his/her  management,  the  users  and  the  sponsors 
of  the  training  system.  The  design  evaluation  of  each  completed 
task  should  be  documented,  noting  problems  which  were  encountered, 
their  solutions,  and  residual  effects.  Efficient  documentation  is 
required  to  avoid  undue  time  loss. 
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DEVELOPMENT  EVALUATION  GUIDELINES 


INTRODUCTION 

Developmental  test  and  evaluation  (DTE)  should  occur  throughout  the 
development  cycle  of  an  AST  training  system  (see  Figure  3,  presented 
earlier  in  Section  V).  As  each  critical  configuration  item  is  developed, 
such  as  the  automated  performance  measurement  software,  it  should  be  tested 
and  evaluated.  Revisions  should  be  made  according  to  the  results.  This 
cycle  of  test  evaluate,  and  revise  (TEAR)  should  occur  regularly  during  the 
development  of  the  system.  It  should  occur  not  only  at  the  level  of 
critical  configuration  items,  but  also  during  integration  of  the  items  into 
the  subsystems,  such  as  instructor  model  and  simulation/event  control. 

Most  importantly,  the  TEAR  cycle  must  be  executed  at  the  final  stage, 
after  the  subsystems  are  integrated  into  the  prototype  training  system. 
The  evaluation  criteria  for  the  TEAR  process  will  change  as  it  is  applied 
at  different  stages  of  system  development.  The  TEAR  process  at  the  final 
integration  stage  will  involve  top-level  criteria  such  as,  what  is  the 
training  effectiveness  of  the  total  system.  A  structure  for  the  various 
levels  of  evaluation  criteria  should  be  generated  before  the  development  is 
begun. 

Recommendations  for  allocation  of  the  responsibility  for  TEAR  is 
beyond  the  scope  of  this  report.  As  an  informal  process,  TEAR  may  occur  on 
a  regular  (daily  or  weekly)  basis  by  the  contractor's  training  system 
development  team.  The  procedures  for  more  formal  DTE  are  promulgated  in  a 
proposed  OPNAV  Instruction  (Cordell,  Nutter  and  Heidt,  1979).  A  Device 
Test  and  Evaluation  Master  Plan  (DTEMP),  as  decribed  in  the  proposed  OPNAV 
Instruction,  is  in  part,  a  vehicle  for  establishing  check  points  for  when 
more  formal  DTE  is  conducted  by  an  independent  evaluation  team.  The  DTI 
followed  by  revision,  is  necessary  to  promote  the  ultimate  achievement,  of 
program  objectives,  i.e.,  an  effective  and  efficient  training  system  that 
enjoys  good  user  acceptance. 

The  TEAR  process  should  examine  whether  the  functional  design 
requirements  have  been  satisfied.  Whenever  possible,  the  TEAR  should 
assess  the  performance  of  each  component  quantitatively.  These  measures 
must  be  evaluated  against  criteria  derived  on  the  basis  of  conformity  with 
the  functional  design  requirements.  For  example,  a  functional  design 
requirement  (or  key  functional  characteristic)  might  state  that  the  speech 
recognition  system  should  be  capable  of  operating  under  ambient  noise 
conditions  equivalent  to  those  found  at  the  intended  site  of  the  training 
system.  The  TEAR  process  would  require,  first,  that  the  noise  at  that  site 
be  measured  under  appropriate  conditions.  Second,  various  noise-cancelling 
microphones  might  be  used  with  the  recognition  system  to  determine  their 
effect  on  recognition  accuracy  in  the  noisy  environment.  This  TEAR  step 
would  produce  confirmation  that  a  particular  recognizer  and  microphone  were 
suitable.  Furthermore,  if  information  about  mic  placement  or  volume 
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adjustment  were  obtained  in  the  TEAR  process,  it  would  be  forwarded  to  the 
courseware  developer  to  become  part  of  the  procedural  instruction  for  the 
trainee. 

In  summary,  the  repeated  TEAR  process,  whether  conducted  informally  by 
the  contractor,  or  formally,  as  a  DTE  functional  configuration  audit,  must 
determine  compliance  with  the  functional  design  requirements.  This 
determination  is  made,  whenever  possible,  by  devising  quantitative  tests 
and  generating  explicit  evaluation  criteria. 

Examples  of  some  of  the  issues  that  might  be  relevant  for  TEAR  in  a 
speech  subsystem  are  given  in  Table  E-l.  This  list  is  not  exhaustive,  but 
serves  merely  to  illustrate  the  level  of  simple  but  important  questions 
that  should  be  confronted  during,  rather  than  after,  system  development. 

The  final  stages  of  system  development  require  the  TEAR  to  be  directed 
toward  the  most  global  evaluation  criteria.  Effective  training  should  be 
the  ultimate  test  of  a  training  system.  Short  of  performing  a  full 
training  effectiveness  evaluation,  the  TEAR  process  can  be  valuable  by 
addressing  issues  such  as: 

curriculum  content 
tutorial  techniques 

instructional  features  (freeze,  slow-motion, 
playback,  etc.) 
performance  measurement 
pacing  of  entire  training  course 

servicing  the  trainee's  perceived  instructional  needs, 
such  as 

-  request  for  instructor 

-  request  review 

-  request  break 

-  request  greater  difficulty 

servicing  the  instructors  needs,  such  as 

-  request  current  trainee  status 

-  request  trainee's  records 

-  request  trainee’s  class  standing 

-  request  diagnosis  of  strength/weaknesses 


A  group  of  SMEs  from  the  operational  training  community  should  be 
involved  in  the  TEAR  process  at  the  final  integration  stage  of  training 
system  development.  Their  opinion  represents  a  valuable  indicator  of 
training  effecti venesss  and  user  acceptance.  Their  comments  and  critique 
should  be  weighted  heavily  in  the  subsequent  (and  critical)  revision. 
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TABLE  E-l.  EXAMPLE  ISSUES  FOR  DEVELOPMENTAL  TEST  EVALUATION  AND 
REVISION  (TEAR)  OF  AN  AUTOMATED  SPEECH  SUBSYSTEM 


Vocabulary  Analysis  and  Speech  Data  Collection  (SDC) 

Does  it  comply  with  the  operational  task? 

(Do  speech  recognition  constraints  compromise 
the  use  of  operational  vocabulary?) 

How  much  time  is  required  for  SDC? 

(assessing  a  speaker-dependent  system) 

Is  the  SDC  structured  to  support  instruction? 

Is  SDC  done  in  context  of  the  task? 

Are  reasonable  alternative  phrasings  allowed? 

Is  instruction  given  on  articulation,  volume,  etc.? 

Speech  Recognition  Performance 

Is  the  recognition  accuracy  adequate  to  support  training? 

Does  the  response  accuracy  of  the  system  approach  99% 
on  phrases  critical  to  the  task,  e.g.,  time-constrai ned 
interactive  events? 

Is  a  Test  Mode  available? 

Is  the  trainee  instructed  on  how  to  improve 
recognition  accuracy? 

Is  the  recognition  system  robust  with  respect  to 
ambient  noise  and  speaker  variabilities? 

Is  the  rejection  rate  acceptable? 

Is  the  time  required  for  recognition  and 
understanding  acceptable? 


(Continued) 
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TABLE  E-l.  (Continued) 

Speech  Generation 

Is  the  voice  easily  understandable? 

Is  the  voice  quality  (inflection,  etc.)  acceptable? 

Are  discriminably  different  voices  available  to 
represent  different  speakers? 

Is  the  time  for  speech  generation  acceptable? 

Is  the  vocabulary  and  its  use  consistent  with 
current  operational  language? 
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DEVELOPMENT  EVALUATION  GUIDELINES 


1.  A  repeated  cycle  of  test,  evaluate,  and  revise  (TEAR)  should  be 
planned  at  the  outset  of  system  development. 

2.  Guidelines  for  formal  developmental  test  and  evaluation  (DTE)  are 
given  in  proposed  OPNAV  Instruction  (Cordell,  et  al . ,  1979). 

3.  The  TEAR  plan  should  be  structured  hierarchically  to  address 
increasingly  larger  functional  units  of  the  training  system  over 
time. 

4.  The  TEAR  process  should  be  based  on  explicit  evaluation  criteria 
appropriate  for  each  stage  of  system  development  and  integration. 

5.  Quantitative  tests  and  evaluation  criteria  should  be  generated 
whenever  possible. 

6.  The  final  stages  of  system  development  and  integration  must  be 
subjected  to  the  TEAR  process.  Tests  at  this  stage  should  use 
trainees  representative  of  the  user  population,  and  should 
exercise  all  branches  of  the  system  courseware.  A  group  of  SMEs 
is  recommended  to  assist  in  this  final  TEAR  cycle  of  the  system 
development. 
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Acoustical  Sciences  Division 
Code  L348 

Pensacola,  FL  32508 

MAJ  Neal  Morgan  (LGY) 

Air  Force  Logistics  Mgmt  Center 
Bldg.  205 

Gunter  AFB,  AL  36114 

Dr.  Michael  G.  Sanders 
US  Army  Res.  Inst.  Field  Unit 
P.O.  Box  476 
Ft.  Rucker,  AL  36362 
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CW02  Ray  Priest 
Naval  Air  Tech.  Training  Center 
Code  7411 
NAS  Memphis  (85) 

Millington,  TN  38054 

Lt.  Col.  Robert  O'Donnell 
6570  ARML-HEA 
Wright- Patterson  AFB 
Dayton,  OH  45322 

Mr.  Eric  Werkowitz 
AFFDL/FGR 

Wright  Patterson  AFB,  OH  45433 
ASD/AXA 

Attn:  N.  R.  Vivians 
Wright-Patterson  AFB,  OH  45433 

CAPT  Barry  P.  McFarland 
US  Air  Force 
ASD/ENECH 

Wright-Patterson  AFB,  OH  45433 

Mr.  Don  F.  McKechnie 
Research  Psychologist 
AFAMRL/HEF 

Wright-Patterson  AFB,  OH  45433 

CAPT  Ronald  J.  Marine 
US  Air  Force 
ASD/AER-EX 

Wright-Patterson  AFB,  OH  45433 

Mr.  John  Courtright 
AFAMRL/HEG 

Wright-Patterson  AFB,  OH  45433 

Mr.  Don  Monk 
AMRL/HED 

Wright-Patterson  AFB,  OH  45433 


CAPT  Vince  Mortimer 
AFAMRL/BBM 

Wright-Patterson  AFB,  OH  45433 

Noel  P.  Schwartz 
AFHRL/ASM 

Advanced  Systems  Division 
Wright-Patterson  AFB,  OH  45433 

Mr.  Timothy  Theis 
ASD/RWR 

Building  20,  Aero.  B 
Wright-Patterson  AFB,  OH  45433 

Chief  of  Naval  Education  &  Training 
Liaison  Office 
Human  Resource  Laboratory 
Flying  Training  Division 
Williams  AFB,  AZ  85224 

Robert  F.  Lawson,  CD R,  USN  (Ret) 
Naval  Applications  Engineer 
ONR  Scientific  Department 
1030  E.  Green  Street 
Pasadena,  CA  91106 

Dr .  Keith  Bromley 

Naval  Ocean  Systems  Center 

Code  8111 

Sam  Diego,  CA  92152 

Mr.  Warren  Lewis 
Naval  Ocean  Systems  Center 
Human  Engineering  Branch 
Code  8231 

San  Diego,  CA  92152 

Mr.  Harry  A.  Whitted 
Code  8235 

Naval  Ocean  Systems  Center 
271  Catalina  Boulevard 
San  Diego,  CA  92152 


Mr.  Thomas  J.  Moore 
AFAMRL/BBA 

Wright-Patterson  AFB,  OH  45433 
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Dr.  Robert  A.  Wisher 
Navy  Personnel  Research  and 
Development  Center 
Code  P309 

San  Diego,  CA  92152 

Mr.  Me  Ivy  n  C.  Moy 
Navy  Personnel  Res.  &  Dev.  Center 
Information  &  Decision  Processes 
Code  305 

San  Diego,  CA  92152 
John  Silva 

Naval  Ocean  Systems  Center 
Code  823 

San  Diego,  CA  92152 

Mr.  Gary  Poock 
Naval  PG  School 
Code '55PK 

Monterey,  CA  93940 

Mr.  Clayton  R.  Coler 
Research  Scientist 
NASA  Ames  Research  Center 
Mail  Stop  239-2 
Moffett  Field,  CA  94035 

Hallie  M.  Funkhouser 
Technical  Assistant 
NASA,  Ames  Research  Center 
Mail  Stop  293-3 
Moffett  Field,  CA  94035 

Kinga  M.  Perlacki 

NASA,  Ames  Research  Center 

Mail  Stop  239-2 

Moffett  Field,  CA  94035 

Dr.  Edward  Huff 

Chief,  Helicopter  Human  Factors  Ofc 
Mail  Stop  239-21 
NASA,  Ames  Research  Center 
Moffett  Field,  CA  94035 


Dr.  Robert  P.  Plummber 

Asst.  Prof. ,  University  of  Utah 

NASA,  Ames  Research  Center 

Mail  Stop  239-2 

Moffett  Field,  CA  94035 

Mr.  Robert  H.  Wright 

Research  Psychologist 

Army  Research  Inst.  Field  Unit 

P.0.  Box  476 

Ft.  Rucker,  AL  56362 

Commander 

Naval  Air  Systems  Command 
Air  413G 

Washington,  DC  20361 

Mr.  Ray  Satterfield 
Photographer 

Naval  Air  Development  Center 
Warminster,  PA  18974 

National  Aviation  Facilities 
Experimental  Center 
Library 

Atlantic  City,  NJ  08405 

Chief  of  Naval  Operations 
OP-593B 

Washington,  DC  20350 

Commander 

Naval  Air  Force 

US  Pacific  Fleet  (Code  316) 

NAS  North  Island 
San  Diego,  CA  92135 

Commander 

Training  Command 

Attn:  Educational  Advisor 

US  Pacific  Fleet 

San  Diego,  CA  92147 
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Commander 

Naval  Weapons  Center 
Code  3154 
Attn:  Mr.  Curtis 
China  Lake,  CA  93555 

Commanding  Officer 
Fleet  Anti-Submarine  Warfare 
Training  Center,  Pacific 
Attn:  Code  001 
San  Diego,  CA  92147 

Commander 

Naval  Air  Test  Center 
CT  176 

Patuxent  River,  MD  20670 

Dr.  J.  D.  Fletcher 
Defense  Adv.  Research 
Projects  Agency  (CTO) 

1400  Wilson  Boulevard 
Arlington,  VA  22209 

Commanding  Officer 

Naval  Air  Technical  Training  Center 

Code  104,  Building  S-54 

NAS  Memphis  (85) 

Millington,  TN  38054 

Mr.  Walt  Primas 

Chief  of  Naval  Operations 

OP-39T 

Washington,  DC  20350 

Chief  of  Naval  Operations 
OP-987H 

Attn:  Dr.  R.G.  Smith 
Washington,  DC  20350 

Commander 

Naval  Air  Systems  Command 
Technical  Library 
AIR-950D 

Washington,  DC  20361 


Commander 

Naval  Air  Force 

US  Pacific  Fleet  (Code  342) 

NAS  North  Island 
Sam  Diego,  CA  92135 

Commander 

Naval  Air  Development  Center 
Attn:  Code  6022 
Warminster,  PA  18974 

Commander 

Naval  Sea  Systems  Command 
Attn:  H.  Baker,  Code  6122 
Washington,  DC  20362 

Navy  Personnel  Research  and 
Development  Center 
Attn:  McDowell 
Library,  Code  P201L 
San  Diego,  CA  92152 

Chief  of  Naval  Operations 
OP- 96 

Washington,  DC  20350 
Commandant 

US  Army  Field  Artillery  School 

ATSF-TD-T 

Mr .  Inman 

Ft.  Sill,  OK  73503 
Commandant 

US  Army  Field  Artillery  School 
Counterfire  Department 
Attn:  Eugene  C.  Rogers 
Ft.  Sill,  OK  73503 
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Director 

US  Army  Human  Eng .  Laboratory 
Attn:  DRXHE-HE  (KEESEE) 

Aberdeen  Proving  Ground,  MD  21005 

USAHEL/USAAVNC 

Attn:  DRXHF-FR  (Dr.  Hoffmann) 

P.0.  Box  476 

Ft.  Rucker,  AL  36362 

ASD/ENESS 

Attn:  R.B.  Kuhnen 

Wright  Patterson  AFB,  OH  45433 

Air  Force  Human  Resources  Lab 
AFHRL/LR 

Logistics  Research  Division 
Wright  Patterson  AFB,  OH  45433 

LT  Dave  Cooper 
AFHRL/OTT 

Wright  Patterson  AFB,  OH  45433 

US  Air  Force  Human  Resources  Lab 
AFHRL-IT  (Dr.  Rockway) 

Technical  Training  Division 
Loury  AFB,  CO  80230 

US  Air  Force  Human  Resources  Lab 
TSZ 

Brooks  AFB,  TX  78235 

ASD/ENETC 

Mr.  R.G.  Cameron 

Wright  Patterson  AFB,  OH  45433 

Headquarters  34  Tactical  Airlift 
Training  Group/TTDI 
Little  Rock  AFB,  AL  ’’2076 

Headquarters 

Air  Training  Command,  XPTI 
Attn :  Mr .  Goldman 
Randolph  AFB,  TX  78148 


Commanding  Officer 

Rome  Air  Development  Center 

Library  (TSLD) 

Griffiss  AFB,  NY  13446 

Director 

Air  University  Library 
Maxwell  AFB,  AL  36100 

Mr.  Harold  A.  Kottmann 
ASD/YWE 

Wright  Patterson  AFB,  OH  45433 

Aeronautical  Systems  Div. 

USAF ,  ASD/YWB  (R.  Coward) 
Wright-Patterson  AFB 
Dayton,  OH  45433 

CDR  Charles  Theisen 
Lauren  Ridge 
R.D.  //2,  Box  143-8A 
New  Hope,  PA  18938 

Chief 

ARI  Field  Unit 

P .  .0.  Box  476 

Ft.  Rucker,  AL  36362 


